Nov 04 04:44:33.251344 ip-10-0-141-132 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 04 04:44:33.251357 ip-10-0-141-132 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 04 04:44:33.251366 ip-10-0-141-132 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 04 04:44:33.251700 ip-10-0-141-132 systemd[1]: Failed to start Kubernetes Kubelet. Nov 04 04:44:44.809954 ip-10-0-141-132 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 04 04:44:44.809971 ip-10-0-141-132 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot afa9e07559f44b5f9c73cb4c19e08c48 -- Nov 04 04:46:40.310329 ip-10-0-141-132 systemd[1]: Starting Kubernetes Kubelet... Nov 04 04:46:40.665336 ip-10-0-141-132 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 04:46:40.665336 ip-10-0-141-132 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 04:46:40.665336 ip-10-0-141-132 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 04:46:40.665336 ip-10-0-141-132 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 04:46:40.668363 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668286 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:46:40.668363 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668363 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:46:40.668363 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668367 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:46:40.668462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668371 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:46:40.668462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668373 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:46:40.668462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668376 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:46:40.668462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668378 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:46:40.668462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668382 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:46:40.668462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668385 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:46:40.668462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668387 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:46:40.668462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668390 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:46:40.668462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668392 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:46:40.668462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668395 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:46:40.668462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668397 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:46:40.668462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668400 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:46:40.668462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668402 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:46:40.668462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668405 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:46:40.668462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668408 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:46:40.668462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668410 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:46:40.668462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668413 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:46:40.668462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668415 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:46:40.668462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668418 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:46:40.668462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668420 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:46:40.668920 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668423 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:46:40.668920 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668425 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:46:40.668920 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668433 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:46:40.668920 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668437 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:46:40.668920 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668439 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:46:40.668920 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668441 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:46:40.668920 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668444 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:46:40.668920 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668446 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:46:40.668920 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668449 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:46:40.668920 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668451 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:46:40.668920 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668454 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:46:40.668920 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668456 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:46:40.668920 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668459 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:46:40.668920 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668462 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:46:40.668920 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668465 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:46:40.668920 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668468 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:46:40.668920 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668470 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:46:40.668920 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668473 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:46:40.668920 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668476 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:46:40.668920 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668478 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:46:40.669412 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668481 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:46:40.669412 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668484 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:46:40.669412 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668486 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:46:40.669412 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668489 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:46:40.669412 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668491 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:46:40.669412 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668493 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:46:40.669412 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668496 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:46:40.669412 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668498 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:46:40.669412 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668501 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:46:40.669412 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668504 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:46:40.669412 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668506 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:46:40.669412 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668508 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:46:40.669412 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668511 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:46:40.669412 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668513 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:46:40.669412 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668516 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:46:40.669412 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668519 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:46:40.669412 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668521 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:46:40.669412 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668524 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:46:40.669412 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668528 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:46:40.669882 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668532 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:46:40.669882 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668535 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:46:40.669882 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668537 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:46:40.669882 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668540 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:46:40.669882 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668542 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:46:40.669882 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668546 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:46:40.669882 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668550 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:46:40.669882 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668552 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:46:40.669882 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668555 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:46:40.669882 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668558 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:46:40.669882 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668560 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:46:40.669882 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668563 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:46:40.669882 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668566 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:46:40.669882 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668568 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:46:40.669882 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668571 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:46:40.669882 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668573 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:46:40.669882 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668576 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:46:40.669882 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668580 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:46:40.669882 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668584 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:46:40.669882 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668587 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:46:40.670372 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668589 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:46:40.670372 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668592 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:46:40.670372 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668594 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:46:40.670372 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668597 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:46:40.670372 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668599 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:46:40.670372 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668602 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:46:40.670372 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668604 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:46:40.670372 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668607 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:46:40.670372 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668609 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:46:40.670372 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668612 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:46:40.670372 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668615 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:46:40.670372 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668618 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:46:40.670372 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668620 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:46:40.670372 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668623 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:46:40.670372 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668626 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:46:40.670372 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668628 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:46:40.670372 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668631 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:46:40.670372 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668633 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:46:40.670372 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668636 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:46:40.670825 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668639 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:46:40.670825 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668642 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:46:40.670825 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668644 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:46:40.670825 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668648 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:46:40.670825 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.668650 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:46:40.670825 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669431 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:46:40.670825 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669438 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:46:40.670825 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669441 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:46:40.670825 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669444 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:46:40.670825 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669446 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:46:40.670825 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669449 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:46:40.670825 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669452 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:46:40.670825 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669454 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:46:40.670825 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669458 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:46:40.670825 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669460 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:46:40.670825 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669463 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:46:40.670825 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669466 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:46:40.670825 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669468 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:46:40.670825 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669471 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:46:40.670825 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669473 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:46:40.671418 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669476 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:46:40.671418 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669479 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:46:40.671418 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669481 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:46:40.671418 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669487 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:46:40.671418 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669490 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:46:40.671418 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669493 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:46:40.671418 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669496 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:46:40.671418 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669499 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:46:40.671418 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669501 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:46:40.671418 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669504 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:46:40.671418 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669506 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:46:40.671418 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669509 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:46:40.671418 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669511 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:46:40.671418 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669514 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:46:40.671418 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669516 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:46:40.671418 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669518 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:46:40.671418 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669521 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:46:40.671418 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669524 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:46:40.671418 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669526 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:46:40.671418 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669529 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:46:40.671881 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669531 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:46:40.671881 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669534 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:46:40.671881 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669536 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:46:40.671881 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669539 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:46:40.671881 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669541 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:46:40.671881 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669544 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:46:40.671881 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669546 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:46:40.671881 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669549 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:46:40.671881 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669551 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:46:40.671881 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669553 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:46:40.671881 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669557 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:46:40.671881 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669561 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:46:40.671881 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669565 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:46:40.671881 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669570 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:46:40.671881 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669573 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:46:40.671881 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669578 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:46:40.671881 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669581 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:46:40.671881 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669584 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:46:40.671881 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669587 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:46:40.672353 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669589 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:46:40.672353 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669592 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:46:40.672353 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669594 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:46:40.672353 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669596 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:46:40.672353 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669599 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:46:40.672353 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669601 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:46:40.672353 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669604 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:46:40.672353 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669606 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:46:40.672353 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669608 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:46:40.672353 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669611 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:46:40.672353 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669613 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:46:40.672353 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669616 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:46:40.672353 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669618 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:46:40.672353 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669620 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:46:40.672353 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669623 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:46:40.672353 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669625 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:46:40.672353 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669628 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:46:40.672353 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669630 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:46:40.672353 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669633 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:46:40.672353 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669635 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:46:40.672848 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669638 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:46:40.672848 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669640 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:46:40.672848 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669642 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:46:40.672848 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669645 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:46:40.672848 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669647 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:46:40.672848 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669650 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:46:40.672848 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669652 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:46:40.672848 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669655 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:46:40.672848 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669660 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:46:40.672848 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669663 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:46:40.672848 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669666 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:46:40.672848 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669669 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:46:40.672848 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669672 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:46:40.672848 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669674 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:46:40.672848 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669677 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:46:40.672848 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669679 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:46:40.672848 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669681 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:46:40.672848 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669684 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:46:40.672848 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669686 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:46:40.673320 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669689 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:46:40.673320 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669692 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:46:40.673320 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669694 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:46:40.673320 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669696 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:46:40.673320 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669699 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:46:40.673320 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669701 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:46:40.673320 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669703 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:46:40.673320 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669707 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:46:40.673320 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669709 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:46:40.673320 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669712 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:46:40.673320 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669714 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:46:40.673320 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669717 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:46:40.673320 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.669719 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:46:40.673320 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670244 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:46:40.673320 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670249 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:46:40.673320 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670252 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:46:40.673320 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670254 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:46:40.673320 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670257 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:46:40.673320 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670260 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:46:40.673320 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670262 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:46:40.673789 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670264 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:46:40.673789 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670268 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:46:40.673789 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670273 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:46:40.673789 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670278 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:46:40.673789 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670281 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:46:40.673789 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670284 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:46:40.673789 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670287 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:46:40.673789 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670289 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:46:40.673789 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670292 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:46:40.673789 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670295 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:46:40.673789 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670297 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:46:40.673789 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670300 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:46:40.673789 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670302 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:46:40.673789 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670305 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:46:40.673789 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670308 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:46:40.673789 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670310 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:46:40.673789 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670312 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:46:40.673789 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670315 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:46:40.673789 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670317 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:46:40.674260 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670320 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:46:40.674260 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670322 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:46:40.674260 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670327 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:46:40.674260 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670330 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:46:40.674260 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670333 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:46:40.674260 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670335 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:46:40.674260 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670338 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:46:40.674260 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670340 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:46:40.674260 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670343 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:46:40.674260 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670345 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:46:40.674260 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670347 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:46:40.674260 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670350 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:46:40.674260 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670353 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:46:40.674260 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670355 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:46:40.674260 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670358 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:46:40.674260 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670361 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:46:40.674260 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670364 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:46:40.674260 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670367 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:46:40.674260 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670369 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:46:40.674696 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670372 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:46:40.674696 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670375 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:46:40.674696 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670377 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:46:40.674696 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670380 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:46:40.674696 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670382 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:46:40.674696 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670385 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:46:40.674696 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670387 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:46:40.674696 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670390 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:46:40.674696 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670392 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:46:40.674696 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670395 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:46:40.674696 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670397 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:46:40.674696 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670400 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:46:40.674696 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670402 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:46:40.674696 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670405 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:46:40.674696 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670407 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:46:40.674696 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670410 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:46:40.674696 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670412 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:46:40.674696 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670415 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:46:40.674696 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670417 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:46:40.674696 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670419 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:46:40.675167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670422 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:46:40.675167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670424 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:46:40.675167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670427 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:46:40.675167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670429 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:46:40.675167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670432 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:46:40.675167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670434 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:46:40.675167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670436 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:46:40.675167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670439 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:46:40.675167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670442 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:46:40.675167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670445 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:46:40.675167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670447 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:46:40.675167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670451 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:46:40.675167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670453 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:46:40.675167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670456 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:46:40.675167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670458 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:46:40.675167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670461 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:46:40.675167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670463 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:46:40.675167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670466 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:46:40.675167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670468 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:46:40.675603 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670471 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:46:40.675603 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670473 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:46:40.675603 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670475 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:46:40.675603 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670478 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:46:40.675603 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670480 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:46:40.675603 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670483 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:46:40.675603 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670486 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:46:40.675603 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670488 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:46:40.675603 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670491 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:46:40.675603 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670493 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:46:40.675603 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670495 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:46:40.675603 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670498 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:46:40.675603 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670500 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:46:40.675603 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670503 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:46:40.675603 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670505 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:46:40.675603 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670508 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:46:40.675603 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670510 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:46:40.675603 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670513 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:46:40.675603 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670515 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:46:40.675603 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670517 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:46:40.676075 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670520 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:46:40.676075 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.670522 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:46:40.676075 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671310 2577 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 04:46:40.676075 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671320 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 04:46:40.676075 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671340 2577 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 04:46:40.676075 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671350 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 04:46:40.676075 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671354 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 04:46:40.676075 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671357 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 04:46:40.676075 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671361 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 04:46:40.676075 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671366 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 04:46:40.676075 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671369 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 04:46:40.676075 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671372 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 04:46:40.676075 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671375 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 04:46:40.676075 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671379 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 04:46:40.676075 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671382 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 04:46:40.676075 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671384 2577 flags.go:64] FLAG: --cgroup-root="" Nov 04 04:46:40.676075 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671387 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 04:46:40.676075 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671390 2577 flags.go:64] FLAG: --client-ca-file="" Nov 04 04:46:40.676075 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671393 2577 flags.go:64] FLAG: --cloud-provider="external" Nov 04 04:46:40.676075 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671395 2577 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 04:46:40.676075 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671402 2577 flags.go:64] FLAG: --cluster-domain="" Nov 04 04:46:40.676075 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671405 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 04:46:40.676075 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671410 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 04:46:40.676597 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671413 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 04:46:40.676597 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671416 2577 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 04:46:40.676597 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671422 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 04:46:40.676597 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671425 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 04:46:40.676597 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671428 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 04:46:40.676597 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671431 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 04:46:40.676597 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671434 2577 flags.go:64] FLAG: --contention-profiling="false" Nov 04 04:46:40.676597 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671437 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 04:46:40.676597 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671440 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 04:46:40.676597 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671443 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 04:46:40.676597 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671446 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 04:46:40.676597 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671450 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 04:46:40.676597 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671453 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 04:46:40.676597 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671456 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 04:46:40.676597 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671459 2577 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 04:46:40.676597 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671467 2577 flags.go:64] FLAG: --enable-server="true" Nov 04 04:46:40.676597 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671470 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 04:46:40.676597 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671477 2577 flags.go:64] FLAG: --event-burst="100" Nov 04 04:46:40.676597 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671480 2577 flags.go:64] FLAG: --event-qps="50" Nov 04 04:46:40.676597 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671483 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 04:46:40.676597 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671486 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 04:46:40.676597 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671489 2577 flags.go:64] FLAG: --eviction-hard="" Nov 04 04:46:40.676597 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671493 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 04:46:40.676597 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671496 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 04:46:40.677220 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671499 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 04:46:40.677220 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671502 2577 flags.go:64] FLAG: --eviction-soft="" Nov 04 04:46:40.677220 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671505 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 04:46:40.677220 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671507 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 04:46:40.677220 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671510 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 04:46:40.677220 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671513 2577 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 04:46:40.677220 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671516 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 04:46:40.677220 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671520 2577 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 04:46:40.677220 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671523 2577 flags.go:64] FLAG: --feature-gates="" Nov 04 04:46:40.677220 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671527 2577 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 04:46:40.677220 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671531 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 04:46:40.677220 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671534 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 04:46:40.677220 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671537 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 04:46:40.677220 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671540 2577 flags.go:64] FLAG: --healthz-port="10248" Nov 04 04:46:40.677220 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671543 2577 flags.go:64] FLAG: --help="false" Nov 04 04:46:40.677220 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671546 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:46:40.677220 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671549 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 04:46:40.677220 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671552 2577 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 04:46:40.677220 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671554 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 04:46:40.677220 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671558 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 04:46:40.677220 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671561 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 04:46:40.677220 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671564 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 04:46:40.677220 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671567 2577 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 04:46:40.677782 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671570 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 04:46:40.677782 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671578 2577 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 04:46:40.677782 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671581 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 04:46:40.677782 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671584 2577 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 04:46:40.677782 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671587 2577 flags.go:64] FLAG: --kube-reserved="" Nov 04 04:46:40.677782 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671589 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 04:46:40.677782 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671592 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 04:46:40.677782 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671595 2577 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 04:46:40.677782 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671598 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 04:46:40.677782 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671601 2577 flags.go:64] FLAG: --lock-file="" Nov 04 04:46:40.677782 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671604 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 04:46:40.677782 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671607 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 04:46:40.677782 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671609 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 04:46:40.677782 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671615 2577 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 04:46:40.677782 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671618 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 04:46:40.677782 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671621 2577 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 04:46:40.677782 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671625 2577 flags.go:64] FLAG: --logging-format="text" Nov 04 04:46:40.677782 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671628 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 04:46:40.677782 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671631 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 04:46:40.677782 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671635 2577 flags.go:64] FLAG: --manifest-url="" Nov 04 04:46:40.677782 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671638 2577 flags.go:64] FLAG: --manifest-url-header="" Nov 04 04:46:40.677782 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671642 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 04:46:40.677782 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671645 2577 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 04:46:40.677782 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671649 2577 flags.go:64] FLAG: --max-pods="110" Nov 04 04:46:40.677782 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671652 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671655 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671657 2577 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671660 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671663 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671666 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671669 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671676 2577 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671679 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671682 2577 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671686 2577 flags.go:64] FLAG: --pod-cidr="" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671689 2577 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671692 2577 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671695 2577 flags.go:64] FLAG: --pods-per-core="0" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671698 2577 flags.go:64] FLAG: --port="10250" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671701 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671704 2577 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-076b0bf1e744ee8f4" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671707 2577 flags.go:64] FLAG: --qos-reserved="" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671710 2577 flags.go:64] FLAG: --read-only-port="10255" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671713 2577 flags.go:64] FLAG: --register-node="true" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671716 2577 flags.go:64] FLAG: --register-with-taints="" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671720 2577 flags.go:64] FLAG: --registry-burst="10" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671723 2577 flags.go:64] FLAG: --registry-qps="5" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671726 2577 flags.go:64] FLAG: --reserved-cpus="" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671729 2577 flags.go:64] FLAG: --reserved-memory="" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671733 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 04:46:40.678426 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671736 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 04:46:40.679032 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671740 2577 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 04:46:40.679032 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671743 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 04:46:40.679032 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671746 2577 flags.go:64] FLAG: --runonce="false" Nov 04 04:46:40.679032 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671749 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 04:46:40.679032 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671752 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 04:46:40.679032 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671755 2577 flags.go:64] FLAG: --seccomp-default="false" Nov 04 04:46:40.679032 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671758 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 04:46:40.679032 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671761 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 04:46:40.679032 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671764 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 04:46:40.679032 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671768 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 04:46:40.679032 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671771 2577 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 04:46:40.679032 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671774 2577 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 04:46:40.679032 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671777 2577 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 04:46:40.679032 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671780 2577 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 04:46:40.679032 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671782 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 04:46:40.679032 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671785 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 04:46:40.679032 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671788 2577 flags.go:64] FLAG: --system-cgroups="" Nov 04 04:46:40.679032 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671792 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 04:46:40.679032 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671797 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 04:46:40.679032 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671800 2577 flags.go:64] FLAG: --tls-cert-file="" Nov 04 04:46:40.679032 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671803 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 04:46:40.679032 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671807 2577 flags.go:64] FLAG: --tls-min-version="" Nov 04 04:46:40.679032 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671810 2577 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 04:46:40.679032 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671813 2577 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 04:46:40.679032 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671815 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 04:46:40.679621 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671818 2577 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 04:46:40.679621 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671821 2577 flags.go:64] FLAG: --v="2" Nov 04 04:46:40.679621 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671826 2577 flags.go:64] FLAG: --version="false" Nov 04 04:46:40.679621 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671830 2577 flags.go:64] FLAG: --vmodule="" Nov 04 04:46:40.679621 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671835 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 04:46:40.679621 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.671838 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 04:46:40.679621 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.671972 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:46:40.679621 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.671978 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:46:40.679621 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.671995 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:46:40.679621 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.671999 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:46:40.679621 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672001 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:46:40.679621 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672004 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:46:40.679621 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672007 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:46:40.679621 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672009 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:46:40.679621 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672012 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:46:40.679621 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672015 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:46:40.679621 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672017 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:46:40.679621 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672020 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:46:40.679621 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672022 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:46:40.679621 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672025 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:46:40.679621 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672028 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:46:40.680128 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672030 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:46:40.680128 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672033 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:46:40.680128 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672035 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:46:40.680128 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672038 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:46:40.680128 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672041 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:46:40.680128 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672043 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:46:40.680128 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672046 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:46:40.680128 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672048 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:46:40.680128 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672051 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:46:40.680128 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672053 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:46:40.680128 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672056 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:46:40.680128 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672058 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:46:40.680128 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672061 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:46:40.680128 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672063 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:46:40.680128 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672065 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:46:40.680128 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672069 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:46:40.680128 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672072 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:46:40.680128 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672075 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:46:40.680128 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672079 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:46:40.680128 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672082 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:46:40.680608 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672084 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:46:40.680608 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672087 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:46:40.680608 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672089 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:46:40.680608 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672092 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:46:40.680608 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672095 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:46:40.680608 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672097 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:46:40.680608 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672100 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:46:40.680608 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672102 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:46:40.680608 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672104 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:46:40.680608 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672107 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:46:40.680608 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672109 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:46:40.680608 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672112 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:46:40.680608 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672114 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:46:40.680608 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672117 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:46:40.680608 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672119 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:46:40.680608 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672121 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:46:40.680608 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672124 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:46:40.680608 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672127 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:46:40.680608 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672129 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:46:40.681118 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672132 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:46:40.681118 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672134 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:46:40.681118 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672137 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:46:40.681118 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672139 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:46:40.681118 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672142 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:46:40.681118 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672144 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:46:40.681118 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672147 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:46:40.681118 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672151 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:46:40.681118 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672156 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:46:40.681118 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672159 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:46:40.681118 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672162 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:46:40.681118 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672166 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:46:40.681118 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672169 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:46:40.681118 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672171 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:46:40.681118 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672174 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:46:40.681118 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672176 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:46:40.681118 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672178 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:46:40.681118 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672181 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:46:40.681118 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672183 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:46:40.681610 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672186 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:46:40.681610 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672188 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:46:40.681610 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672191 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:46:40.681610 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672193 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:46:40.681610 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672196 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:46:40.681610 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672198 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:46:40.681610 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672201 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:46:40.681610 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672203 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:46:40.681610 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672206 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:46:40.681610 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672208 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:46:40.681610 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672210 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:46:40.681610 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672213 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:46:40.681610 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672216 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:46:40.681610 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672218 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:46:40.681610 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672221 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:46:40.681610 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672223 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:46:40.681610 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672225 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:46:40.681610 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672228 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:46:40.681610 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672230 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:46:40.681610 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672233 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:46:40.682100 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672235 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:46:40.682100 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672239 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:46:40.682100 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672241 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:46:40.682100 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672244 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:46:40.682100 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672249 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:46:40.682100 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672251 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:46:40.682100 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672254 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:46:40.682100 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672256 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:46:40.682100 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672258 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:46:40.682100 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672261 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:46:40.682100 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672263 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:46:40.682100 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672266 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:46:40.682100 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672268 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:46:40.682100 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.672295 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 04:46:40.682100 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672330 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:46:40.682100 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672334 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:46:40.682100 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672337 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:46:40.682504 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672341 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:46:40.682504 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672343 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:46:40.682504 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672346 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:46:40.682504 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672348 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:46:40.682504 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672351 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:46:40.682504 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672354 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:46:40.682504 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672356 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:46:40.682504 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672359 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:46:40.682504 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672362 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:46:40.682504 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672365 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:46:40.682504 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672368 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:46:40.682504 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672371 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:46:40.682504 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672373 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:46:40.682504 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672376 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:46:40.682504 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672378 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:46:40.682504 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672380 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:46:40.682504 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672383 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:46:40.682504 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672387 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:46:40.682504 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672389 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:46:40.682941 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672393 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:46:40.682941 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672395 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:46:40.682941 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672398 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:46:40.682941 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672400 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:46:40.682941 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672403 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:46:40.682941 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672405 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:46:40.682941 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672408 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:46:40.682941 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672410 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:46:40.682941 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672412 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:46:40.682941 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672415 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:46:40.682941 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672417 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:46:40.682941 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672420 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:46:40.682941 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672423 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:46:40.682941 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672425 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:46:40.682941 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672428 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:46:40.682941 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672430 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:46:40.682941 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672433 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:46:40.682941 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672435 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:46:40.682941 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672438 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:46:40.683404 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672442 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:46:40.683404 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672446 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:46:40.683404 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672449 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:46:40.683404 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672452 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:46:40.683404 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672455 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:46:40.683404 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672457 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:46:40.683404 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672460 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:46:40.683404 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672462 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:46:40.683404 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672464 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:46:40.683404 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672467 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:46:40.683404 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672469 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:46:40.683404 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672473 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:46:40.683404 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672476 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:46:40.683404 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672480 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:46:40.683404 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672483 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:46:40.683404 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672485 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:46:40.683404 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672487 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:46:40.683404 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672490 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:46:40.683404 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672492 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:46:40.683867 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672495 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:46:40.683867 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672498 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:46:40.683867 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672500 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:46:40.683867 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672502 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:46:40.683867 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672505 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:46:40.683867 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672507 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:46:40.683867 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672510 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:46:40.683867 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672512 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:46:40.683867 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672515 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:46:40.683867 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672517 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:46:40.683867 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672520 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:46:40.683867 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672522 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:46:40.683867 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672525 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:46:40.683867 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672527 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:46:40.683867 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672530 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:46:40.683867 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672533 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:46:40.683867 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672536 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:46:40.683867 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672538 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:46:40.683867 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672541 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:46:40.683867 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672543 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:46:40.684343 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672545 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:46:40.684343 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672548 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:46:40.684343 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672550 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:46:40.684343 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672553 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:46:40.684343 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672556 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:46:40.684343 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672559 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:46:40.684343 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672565 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:46:40.684343 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672569 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:46:40.684343 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672572 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:46:40.684343 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672575 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:46:40.684343 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672578 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:46:40.684343 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672581 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:46:40.684343 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672584 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:46:40.684343 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672586 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:46:40.684343 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672589 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:46:40.684343 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672591 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:46:40.684343 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672594 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:46:40.684343 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672596 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:46:40.684343 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672598 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:46:40.684802 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672601 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:46:40.684802 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672604 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:46:40.684802 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672606 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:46:40.684802 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672609 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:46:40.684802 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672612 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:46:40.684802 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672614 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:46:40.684802 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.672617 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:46:40.684802 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.679398 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 04:46:40.684802 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.679414 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 04:46:40.684802 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.680010 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 04:46:40.684802 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.680017 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 04:46:40.684802 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680078 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:46:40.684802 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680083 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:46:40.684802 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680086 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:46:40.684802 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680089 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:46:40.684802 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680092 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:46:40.684802 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680095 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:46:40.684802 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680097 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:46:40.684802 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680100 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:46:40.684802 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680103 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:46:40.685382 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680106 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:46:40.685382 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680109 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:46:40.685382 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680111 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:46:40.685382 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680114 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:46:40.685382 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680116 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:46:40.685382 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680119 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:46:40.685382 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680121 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:46:40.685382 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680124 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:46:40.685382 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680127 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:46:40.685382 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680130 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:46:40.685382 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680132 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:46:40.685382 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680135 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:46:40.685382 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680138 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:46:40.685382 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680141 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:46:40.685382 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680144 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:46:40.685382 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680146 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:46:40.685382 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680150 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:46:40.685382 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680154 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:46:40.685382 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680157 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:46:40.685382 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680161 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:46:40.686159 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680164 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:46:40.686159 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680166 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:46:40.686159 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680170 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:46:40.686159 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680173 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:46:40.686159 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680175 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:46:40.686159 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680178 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:46:40.686159 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680180 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:46:40.686159 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680183 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:46:40.686159 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680185 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:46:40.686159 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680188 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:46:40.686159 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680190 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:46:40.686159 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680193 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:46:40.686159 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680195 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:46:40.686159 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680198 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:46:40.686159 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680200 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:46:40.686159 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680203 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:46:40.686159 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680206 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:46:40.686159 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680208 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:46:40.686159 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680211 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:46:40.686628 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680213 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:46:40.686628 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680216 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:46:40.686628 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680218 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:46:40.686628 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680221 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:46:40.686628 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680224 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:46:40.686628 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680227 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:46:40.686628 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680229 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:46:40.686628 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680232 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:46:40.686628 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680234 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:46:40.686628 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680237 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:46:40.686628 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680240 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:46:40.686628 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680243 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:46:40.686628 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680246 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:46:40.686628 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680248 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:46:40.686628 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680251 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:46:40.686628 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680253 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:46:40.686628 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680257 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:46:40.686628 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680260 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:46:40.686628 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680262 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:46:40.687140 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680265 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:46:40.687140 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680268 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:46:40.687140 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680270 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:46:40.687140 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680272 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:46:40.687140 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680276 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:46:40.687140 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680280 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:46:40.687140 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680283 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:46:40.687140 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680286 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:46:40.687140 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680288 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:46:40.687140 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680290 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:46:40.687140 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680293 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:46:40.687140 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680295 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:46:40.687140 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680298 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:46:40.687140 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680300 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:46:40.687140 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680303 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:46:40.687140 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680305 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:46:40.687140 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680307 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:46:40.687140 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680310 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:46:40.687140 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680313 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:46:40.687606 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680316 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:46:40.687606 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680318 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:46:40.687606 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680320 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:46:40.687606 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680323 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:46:40.687606 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680325 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:46:40.687606 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680328 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:46:40.687606 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680330 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:46:40.687606 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680333 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:46:40.687606 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680335 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:46:40.687606 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680338 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:46:40.687606 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680340 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:46:40.687606 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680343 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:46:40.687606 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680346 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:46:40.687606 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680348 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:46:40.687606 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680351 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:46:40.687606 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680353 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:46:40.687606 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680355 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:46:40.687606 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680358 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:46:40.687606 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680360 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:46:40.688082 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680362 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:46:40.688082 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.680390 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 04:46:40.688082 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680527 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:46:40.688082 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680531 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:46:40.688082 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680535 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:46:40.688082 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680537 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:46:40.688082 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680540 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:46:40.688082 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680543 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:46:40.688082 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680546 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:46:40.688082 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680548 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:46:40.688082 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680550 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:46:40.688082 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680553 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:46:40.688082 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680556 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:46:40.688082 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680559 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:46:40.688082 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680561 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:46:40.688082 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680564 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:46:40.688462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680566 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:46:40.688462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680569 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:46:40.688462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680571 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:46:40.688462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680574 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:46:40.688462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680577 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:46:40.688462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680579 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:46:40.688462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680581 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:46:40.688462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680584 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:46:40.688462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680587 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:46:40.688462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680590 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:46:40.688462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680592 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:46:40.688462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680595 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:46:40.688462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680597 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:46:40.688462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680600 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:46:40.688462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680602 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:46:40.688462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680604 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:46:40.688462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680608 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:46:40.688462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680612 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:46:40.688462 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680615 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:46:40.688914 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680618 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:46:40.688914 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680620 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:46:40.688914 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680623 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:46:40.688914 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680626 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:46:40.688914 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680628 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:46:40.688914 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680630 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:46:40.688914 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680633 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:46:40.688914 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680635 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:46:40.688914 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680638 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:46:40.688914 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680641 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:46:40.688914 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680643 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:46:40.688914 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680646 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:46:40.688914 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680648 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:46:40.688914 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680651 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:46:40.688914 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680653 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:46:40.688914 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680655 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:46:40.688914 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680658 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:46:40.688914 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680660 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:46:40.688914 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680663 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:46:40.688914 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680665 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:46:40.689402 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680668 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:46:40.689402 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680670 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:46:40.689402 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680672 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:46:40.689402 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680675 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:46:40.689402 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680678 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:46:40.689402 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680681 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:46:40.689402 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680683 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:46:40.689402 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680685 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:46:40.689402 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680688 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:46:40.689402 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680691 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:46:40.689402 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680693 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:46:40.689402 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680696 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:46:40.689402 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680699 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:46:40.689402 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680701 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:46:40.689402 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680704 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:46:40.689402 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680706 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:46:40.689402 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680709 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:46:40.689402 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680711 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:46:40.689402 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680713 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:46:40.689402 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680716 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:46:40.689884 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680719 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:46:40.689884 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680721 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:46:40.689884 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680724 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:46:40.689884 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680727 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:46:40.689884 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680729 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:46:40.689884 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680731 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:46:40.689884 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680734 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:46:40.689884 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680736 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:46:40.689884 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680738 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:46:40.689884 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680741 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:46:40.689884 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680743 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:46:40.689884 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680746 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:46:40.689884 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680748 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:46:40.689884 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680750 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:46:40.689884 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680753 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:46:40.689884 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680755 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:46:40.689884 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680758 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:46:40.689884 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680761 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:46:40.689884 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680764 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:46:40.690344 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680766 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:46:40.690344 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680768 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:46:40.690344 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680771 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:46:40.690344 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680773 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:46:40.690344 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680776 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:46:40.690344 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680779 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:46:40.690344 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680781 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:46:40.690344 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680783 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:46:40.690344 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680787 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:46:40.690344 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680791 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:46:40.690344 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680793 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:46:40.690344 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680796 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:46:40.690344 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680799 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:46:40.690344 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680801 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:46:40.690344 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.680825 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 04:46:40.690344 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680863 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:46:40.690720 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680867 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:46:40.690720 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680870 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:46:40.690720 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680874 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:46:40.690720 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680877 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:46:40.690720 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680879 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:46:40.690720 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680882 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:46:40.690720 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680884 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:46:40.690720 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680887 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:46:40.690720 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680890 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:46:40.690720 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680892 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:46:40.690720 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680895 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:46:40.690720 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680897 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:46:40.690720 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680899 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:46:40.690720 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680902 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:46:40.690720 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680905 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:46:40.690720 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680907 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:46:40.690720 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680910 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:46:40.690720 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680912 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:46:40.690720 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680915 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:46:40.691167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680917 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:46:40.691167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680922 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:46:40.691167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680925 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:46:40.691167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680927 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:46:40.691167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680930 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:46:40.691167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680933 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:46:40.691167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680935 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:46:40.691167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680938 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:46:40.691167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680940 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:46:40.691167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680943 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:46:40.691167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680945 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:46:40.691167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680947 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:46:40.691167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680950 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:46:40.691167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680953 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:46:40.691167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680955 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:46:40.691167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680958 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:46:40.691167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680960 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:46:40.691167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680962 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:46:40.691167 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680966 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:46:40.691602 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680970 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:46:40.691602 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680973 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:46:40.691602 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680975 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:46:40.691602 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680978 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:46:40.691602 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.680981 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:46:40.691602 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681002 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:46:40.691602 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681005 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:46:40.691602 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681008 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:46:40.691602 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681010 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:46:40.691602 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681014 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:46:40.691602 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681016 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:46:40.691602 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681020 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:46:40.691602 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681023 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:46:40.691602 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681026 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:46:40.691602 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681030 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:46:40.691602 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681032 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:46:40.691602 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681035 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:46:40.691602 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681037 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:46:40.691602 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681040 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:46:40.691602 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681043 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:46:40.692203 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681045 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:46:40.692203 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681048 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:46:40.692203 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681051 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:46:40.692203 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681053 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:46:40.692203 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681056 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:46:40.692203 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681059 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:46:40.692203 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681062 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:46:40.692203 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681065 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:46:40.692203 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681067 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:46:40.692203 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681070 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:46:40.692203 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681072 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:46:40.692203 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681075 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:46:40.692203 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681077 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:46:40.692203 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681079 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:46:40.692203 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681082 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:46:40.692203 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681085 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:46:40.692203 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681087 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:46:40.692203 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681090 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:46:40.692203 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681092 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:46:40.692203 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681095 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:46:40.692653 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681097 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:46:40.692653 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681101 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:46:40.692653 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681104 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:46:40.692653 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681107 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:46:40.692653 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681109 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:46:40.692653 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681112 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:46:40.692653 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681116 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:46:40.692653 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681118 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:46:40.692653 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681121 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:46:40.692653 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681124 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:46:40.692653 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681127 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:46:40.692653 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681130 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:46:40.692653 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681132 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:46:40.692653 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681135 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:46:40.692653 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681138 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:46:40.692653 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681140 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:46:40.692653 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681143 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:46:40.692653 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681145 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:46:40.692653 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681148 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:46:40.693115 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681150 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:46:40.693115 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681153 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:46:40.693115 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681155 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:46:40.693115 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681158 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:46:40.693115 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681161 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:46:40.693115 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681163 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:46:40.693115 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681166 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:46:40.693115 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:40.681168 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:46:40.693115 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.681217 2577 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 04:46:40.693115 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.683482 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 04 04:46:40.693115 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.684309 2577 server.go:1014] "Starting client certificate rotation" Nov 04 04:46:40.693115 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.684404 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 04:46:40.693115 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.685264 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 04:46:40.693115 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.687883 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 04:46:40.693115 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.689573 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 04:46:40.698344 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.698327 2577 log.go:25] "Validated CRI v1 runtime API" Nov 04 04:46:40.701946 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.701928 2577 log.go:25] "Validated CRI v1 image API" Nov 04 04:46:40.702844 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.702818 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 04:46:40.705792 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.705771 2577 fs.go:135] Filesystem UUIDs: map[0be9cc94-3840-4d67-9174-653735a4dee2:/dev/nvme0n1p3 436e5e68-db6c-489d-831d-97a4a007cecb:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 04 04:46:40.705868 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.705789 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 04 04:46:40.711746 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.711636 2577 manager.go:217] Machine: {Timestamp:2026-11-04 04:46:40.709390835 +0000 UTC m=+0.294659560 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102727 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22db75a07c4752562af06c9ea31ad4 SystemUUID:ec22db75-a07c-4752-562a-f06c9ea31ad4 BootID:afa9e075-59f4-4b5f-9c73-cb4c19e08c48 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5a:67:1b:8e:a3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5a:67:1b:8e:a3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:f0:dd:d4:b9:45 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 04:46:40.711746 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.711749 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 04:46:40.711868 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.711858 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 04:46:40.714980 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.714953 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 04:46:40.715137 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.714997 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-132.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 04:46:40.715184 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.715140 2577 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 04:46:40.715184 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.715147 2577 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 04:46:40.715184 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.715183 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 04:46:40.715261 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.715195 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 04:46:40.715965 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.715956 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 04:46:40.719350 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.719339 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 04:46:40.719470 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.719462 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 04:46:40.720085 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.720075 2577 kubelet.go:493] "Attempting to sync node with API server" Nov 04 04:46:40.720120 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.720092 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 04:46:40.720120 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.720104 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 04:46:40.720120 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.720112 2577 kubelet.go:405] "Adding apiserver pod source" Nov 04 04:46:40.720193 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.720121 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 04:46:40.720458 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.720446 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 04:46:40.723418 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.723390 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 04:46:40.728219 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.728200 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 04:46:40.729020 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.728967 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 04:46:40.729110 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.729031 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 04:46:40.729250 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.729237 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 04:46:40.729309 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.729258 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 04:46:40.729309 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.729266 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 04:46:40.729309 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.729275 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 04:46:40.729309 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.729284 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 04:46:40.729309 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.729293 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 04:46:40.729830 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.729811 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 04:46:40.729830 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.729827 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 04:46:40.729936 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.729837 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 04:46:40.729936 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.729845 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 04:46:40.729936 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.729854 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 04:46:40.729936 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.729863 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 04:46:40.731207 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.731196 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 04:46:40.731242 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.731211 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 04:46:40.734932 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.734913 2577 server.go:1263] "Started kubelet" Nov 04 04:46:40.735084 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.735014 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 04:46:40.735084 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.735018 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 04:46:40.735217 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.735099 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 04:46:40.735765 ip-10-0-141-132 systemd[1]: Started Kubernetes Kubelet. Nov 04 04:46:40.736050 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.736031 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 04:46:40.737150 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.737136 2577 server.go:317] "Adding debug handlers to kubelet server" Nov 04 04:46:40.739737 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:40.739717 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 04 04:46:40.740754 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.740734 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 04:46:40.741188 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.741170 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 04:46:40.741262 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:40.741246 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-132.us-west-1.compute.internal\" not found" Nov 04 04:46:40.741792 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.741773 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 04:46:40.741792 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.741792 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 04:46:40.741921 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.741773 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 04:46:40.741921 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.741864 2577 reconstruct.go:97] "Volume reconstruction finished" Nov 04 04:46:40.741921 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.741872 2577 reconciler.go:29] "Reconciler: start to sync state" Nov 04 04:46:40.742670 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.742657 2577 factory.go:153] Registering CRI-O factory Nov 04 04:46:40.742729 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.742690 2577 factory.go:223] Registration of the crio container factory successfully Nov 04 04:46:40.742770 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.742741 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 04:46:40.742770 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.742748 2577 factory.go:55] Registering systemd factory Nov 04 04:46:40.742770 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.742753 2577 factory.go:223] Registration of the systemd container factory successfully Nov 04 04:46:40.742770 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.742766 2577 factory.go:103] Registering Raw factory Nov 04 04:46:40.742949 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.742775 2577 manager.go:1199] Started watching for new ooms in manager Nov 04 04:46:40.743516 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.743504 2577 manager.go:319] Starting recovery of all containers Nov 04 04:46:40.755302 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.755284 2577 manager.go:324] Recovery completed Nov 04 04:46:40.759221 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.759209 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:46:40.760746 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.760722 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:46:40.760812 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.760760 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:46:40.760812 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.760770 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:46:40.761286 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.761273 2577 cpu_manager.go:226] "Starting" policy="none" Nov 04 04:46:40.761333 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.761285 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 04:46:40.761333 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.761301 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 04:46:40.761390 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.761351 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:46:40.762024 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.762008 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:46:40.762094 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.762043 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:46:40.762094 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.762061 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:46:40.763295 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.763280 2577 policy_none.go:50] "Start" Nov 04 04:46:40.763295 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.763296 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 04:46:40.763385 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.763310 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 04:46:40.765126 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.765116 2577 policy_none.go:44] "Start" Nov 04 04:46:40.826258 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.812286 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 04:46:40.826258 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.817401 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 04:46:40.826258 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.818866 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 04:46:40.826258 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.818883 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 04:46:40.826258 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.818901 2577 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 04:46:40.826258 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:40.818928 2577 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 04 04:46:40.826484 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.826371 2577 manager.go:341] "Starting Device Plugin manager" Nov 04 04:46:40.826484 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:40.826404 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 04:46:40.826484 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.826413 2577 server.go:87] "Starting device plugin registration server" Nov 04 04:46:40.826648 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.826638 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 04:46:40.826679 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.826650 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 04:46:40.826761 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.826747 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 04:46:40.827047 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.826818 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 04:46:40.827165 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.827050 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 04:46:40.827344 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:40.827327 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 04:46:40.827416 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:40.827377 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-132.us-west-1.compute.internal\" not found" Nov 04 04:46:40.877391 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.877371 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-132.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 04 04:46:40.878214 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:40.877366 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-132.us-west-1.compute.internal.18e4be210b685d63 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-132.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-132.us-west-1.compute.internal,},FirstTimestamp:2026-11-04 04:46:40.734887267 +0000 UTC m=+0.320155992,LastTimestamp:2026-11-04 04:46:40.734887267 +0000 UTC m=+0.320155992,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-132.us-west-1.compute.internal,}" Nov 04 04:46:40.919474 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.919422 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal"] Nov 04 04:46:40.919542 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.919477 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:46:40.920299 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.920283 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:46:40.920348 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.920315 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:46:40.920348 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.920331 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:46:40.922479 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.922468 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:46:40.922630 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.922615 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:46:40.922667 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.922648 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:46:40.923251 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.923232 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:46:40.923303 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.923276 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:46:40.923303 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.923295 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:46:40.923641 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.923621 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:46:40.923730 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.923653 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:46:40.923730 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.923663 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:46:40.925754 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.925738 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:46:40.925810 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.925763 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:46:40.926466 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.926448 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:46:40.926558 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.926483 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:46:40.926558 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.926495 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:46:40.926706 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:40.926687 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-132.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 04 04:46:40.926775 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.926751 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:46:40.927383 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.927368 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:46:40.927454 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.927394 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:46:40.927454 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.927406 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:46:40.927454 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:40.927423 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:46:40.952417 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:40.952395 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-132.us-west-1.compute.internal\" not found" node="ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:46:40.955106 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:40.955089 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-132.us-west-1.compute.internal\" not found" node="ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:46:40.975530 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:40.975503 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-132.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 04 04:46:40.975617 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:40.975475 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-132.us-west-1.compute.internal.18e4be210cf2ef12 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-132.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-132.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-132.us-west-1.compute.internal,},FirstTimestamp:2026-11-04 04:46:40.760745746 +0000 UTC m=+0.346014470,LastTimestamp:2026-11-04 04:46:40.760745746 +0000 UTC m=+0.346014470,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-132.us-west-1.compute.internal,}" Nov 04 04:46:40.975617 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:40.975572 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 04 04:46:41.006482 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:41.006461 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 04:46:41.006585 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:41.006514 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v7wbg" Nov 04 04:46:41.032996 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:41.032966 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 04 04:46:41.033093 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:41.032973 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 04 04:46:41.034061 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:41.033019 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-132.us-west-1.compute.internal.18e4be210cf33ab3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-132.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-132.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-132.us-west-1.compute.internal,},FirstTimestamp:2026-11-04 04:46:40.760765107 +0000 UTC m=+0.346033832,LastTimestamp:2026-11-04 04:46:40.760765107 +0000 UTC m=+0.346033832,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-132.us-west-1.compute.internal,}" Nov 04 04:46:41.042924 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:41.042906 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/81b6936b6f0cd5f6748dab38c3bd34d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal\" (UID: \"81b6936b6f0cd5f6748dab38c3bd34d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:46:41.043035 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:41.042936 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81b6936b6f0cd5f6748dab38c3bd34d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal\" (UID: \"81b6936b6f0cd5f6748dab38c3bd34d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:46:41.043035 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:41.042977 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e7859ee47f36ca521b3b7f587511c5ea-config\") pod \"kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal\" (UID: \"e7859ee47f36ca521b3b7f587511c5ea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:46:41.130325 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:41.130303 2577 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-141-132.us-west-1.compute.internal\" not found" node="ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:46:41.143673 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:41.143653 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/81b6936b6f0cd5f6748dab38c3bd34d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal\" (UID: \"81b6936b6f0cd5f6748dab38c3bd34d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:46:41.143748 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:41.143680 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81b6936b6f0cd5f6748dab38c3bd34d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal\" (UID: \"81b6936b6f0cd5f6748dab38c3bd34d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:46:41.143748 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:41.143735 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/81b6936b6f0cd5f6748dab38c3bd34d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal\" (UID: \"81b6936b6f0cd5f6748dab38c3bd34d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:46:41.143814 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:41.143746 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e7859ee47f36ca521b3b7f587511c5ea-config\") pod \"kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal\" (UID: \"e7859ee47f36ca521b3b7f587511c5ea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:46:41.143814 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:41.143719 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81b6936b6f0cd5f6748dab38c3bd34d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal\" (UID: \"81b6936b6f0cd5f6748dab38c3bd34d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:46:41.143814 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:41.143776 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e7859ee47f36ca521b3b7f587511c5ea-config\") pod \"kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal\" (UID: \"e7859ee47f36ca521b3b7f587511c5ea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:46:41.230160 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:41.230100 2577 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-132.us-west-1.compute.internal\" not found" node="ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:46:41.255569 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:41.255549 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:46:41.258467 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:41.258452 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:46:41.330901 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:41.330876 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:46:41.331830 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:41.331814 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:46:41.331885 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:41.331845 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:46:41.331885 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:41.331855 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:46:41.331885 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:41.331881 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:46:41.388613 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:41.388592 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:46:41.388719 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:41.388618 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-132.us-west-1.compute.internal\": node \"ip-10-0-141-132.us-west-1.compute.internal\" not found" Nov 04 04:46:41.504347 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:41.504273 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-132.us-west-1.compute.internal\" not found" Nov 04 04:46:41.605298 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:41.605271 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-132.us-west-1.compute.internal\" not found" Nov 04 04:46:41.684434 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:41.684414 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 04 04:46:41.684852 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:41.684537 2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ad7f0a69d1aeb4fbea22a6fe43d5cdd1-3a8265abc582d153.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-141-132.us-west-1.compute.internal.18e4be210cf2ef12\": read tcp 10.0.141.132:52256->3.132.101.194:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-141-132.us-west-1.compute.internal.18e4be210cf2ef12 default 5366 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-132.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-132.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-132.us-west-1.compute.internal,},FirstTimestamp:2026-11-04 04:46:40 +0000 UTC,LastTimestamp:2026-11-04 04:46:40.923253982 +0000 UTC m=+0.508522725,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-132.us-west-1.compute.internal,}" Nov 04 04:46:41.705704 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:41.705686 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-132.us-west-1.compute.internal\" not found" Nov 04 04:46:41.806458 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:41.806393 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-132.us-west-1.compute.internal\" not found" Nov 04 04:46:41.907111 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:41.907082 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-132.us-west-1.compute.internal\" not found" Nov 04 04:46:42.007970 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:42.007941 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-132.us-west-1.compute.internal\" not found" Nov 04 04:46:42.011114 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.011081 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 04:41:41 +0000 UTC" deadline="2028-05-08 16:36:00.678111485 +0000 UTC" Nov 04 04:46:42.011173 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.011115 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13235h49m18.666999153s" Nov 04 04:46:42.033774 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.033753 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:46:42.034663 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.034647 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:46:42.034750 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.034686 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:46:42.034750 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.034696 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:46:42.083228 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.083169 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:46:42.134148 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.134127 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:46:42.135041 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.135025 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:46:42.142126 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.142110 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:46:42.201329 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.201305 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 04:46:42.202974 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.202960 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:46:42.260110 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.260086 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 04:46:42.411150 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:42.411118 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7859ee47f36ca521b3b7f587511c5ea.slice/crio-5c1e06c23ceafefa52bd98082bbab3815560178a5df8c33c4d1d034e70ea9f5e WatchSource:0}: Error finding container 5c1e06c23ceafefa52bd98082bbab3815560178a5df8c33c4d1d034e70ea9f5e: Status 404 returned error can't find the container with id 5c1e06c23ceafefa52bd98082bbab3815560178a5df8c33c4d1d034e70ea9f5e Nov 04 04:46:42.411553 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:42.411533 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b6936b6f0cd5f6748dab38c3bd34d9.slice/crio-f3e092af629e4a957bf5377e9721b5f7a1302fdbdb8feba985b3f5fe090cca6d WatchSource:0}: Error finding container f3e092af629e4a957bf5377e9721b5f7a1302fdbdb8feba985b3f5fe090cca6d: Status 404 returned error can't find the container with id f3e092af629e4a957bf5377e9721b5f7a1302fdbdb8feba985b3f5fe090cca6d Nov 04 04:46:42.416442 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.416425 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 04:46:42.564316 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.564282 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:46:42.723320 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.723240 2577 apiserver.go:52] "Watching apiserver" Nov 04 04:46:42.741562 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.741533 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 04 04:46:42.823686 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.823634 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal" event={"ID":"e7859ee47f36ca521b3b7f587511c5ea","Type":"ContainerStarted","Data":"5c1e06c23ceafefa52bd98082bbab3815560178a5df8c33c4d1d034e70ea9f5e"} Nov 04 04:46:42.824769 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.824743 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" event={"ID":"81b6936b6f0cd5f6748dab38c3bd34d9","Type":"ContainerStarted","Data":"f3e092af629e4a957bf5377e9721b5f7a1302fdbdb8feba985b3f5fe090cca6d"} Nov 04 04:46:42.834784 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.834765 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 04:46:42.835124 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.835097 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-znffl","kube-system/kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw","kube-system/global-pull-secret-syncer-m5khv","openshift-cluster-node-tuning-operator/tuned-hlw45","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal","kube-system/konnectivity-agent-7csxg","openshift-dns/node-resolver-wwxmc","openshift-image-registry/node-ca-lfjtg","openshift-multus/multus-cxp8f","openshift-ovn-kubernetes/ovnkube-node-l58zg","openshift-multus/multus-additional-cni-plugins-7rkbf","openshift-multus/network-metrics-daemon-pbjtk","openshift-network-diagnostics/network-check-target-6xc6w"] Nov 04 04:46:42.839141 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.839126 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7csxg" Nov 04 04:46:42.841397 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.841376 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:46:42.841498 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:42.841462 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbjtk" podUID="12f1a157-ce5b-4040-98de-ed53507268db" Nov 04 04:46:42.843615 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.843594 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:46:42.845937 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.845909 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:46:42.848983 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:42.846365 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5khv" podUID="24bc4691-e47f-4f98-ad47-f14654d95d28" Nov 04 04:46:42.849188 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.849169 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:42.850948 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.850928 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:46:42.851048 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:42.851027 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xc6w" podUID="d0e2510f-0579-4594-bd3b-a3dadf3fb81a" Nov 04 04:46:42.853155 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.853135 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lfjtg" Nov 04 04:46:42.854190 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.854166 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/030a8f10-c4b1-45b2-abb8-88ed48e53a80-agent-certs\") pod \"konnectivity-agent-7csxg\" (UID: \"030a8f10-c4b1-45b2-abb8-88ed48e53a80\") " pod="kube-system/konnectivity-agent-7csxg" Nov 04 04:46:42.854270 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.854204 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f1a157-ce5b-4040-98de-ed53507268db-metrics-certs\") pod \"network-metrics-daemon-pbjtk\" (UID: \"12f1a157-ce5b-4040-98de-ed53507268db\") " pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:46:42.854270 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.854232 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98rdv\" (UniqueName: \"kubernetes.io/projected/12f1a157-ce5b-4040-98de-ed53507268db-kube-api-access-98rdv\") pod \"network-metrics-daemon-pbjtk\" (UID: \"12f1a157-ce5b-4040-98de-ed53507268db\") " pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:46:42.854365 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.854277 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:46:42.854365 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.854318 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-socket-dir\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:46:42.854365 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.854345 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-registration-dir\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:46:42.854467 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.854396 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/030a8f10-c4b1-45b2-abb8-88ed48e53a80-konnectivity-ca\") pod \"konnectivity-agent-7csxg\" (UID: \"030a8f10-c4b1-45b2-abb8-88ed48e53a80\") " pod="kube-system/konnectivity-agent-7csxg" Nov 04 04:46:42.854467 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.854441 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-device-dir\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:46:42.854529 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.854471 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-etc-selinux\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:46:42.854529 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.854490 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-sys-fs\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:46:42.854529 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.854517 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd2j4\" (UniqueName: \"kubernetes.io/projected/26237d52-9119-41f2-919a-5e5153ea223a-kube-api-access-nd2j4\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:46:42.855514 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.855497 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-znffl" Nov 04 04:46:42.857773 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.857755 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wwxmc" Nov 04 04:46:42.860056 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.860036 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cxp8f" Nov 04 04:46:42.862606 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.862586 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:42.864941 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.864924 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:46:42.895651 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.895629 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jqdfn\"" Nov 04 04:46:42.895737 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.895662 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 04:46:42.895737 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.895679 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 04:46:42.899766 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.899750 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 04:46:42.900068 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.899987 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 04:46:42.900068 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.900008 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 04:46:42.900207 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.900168 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g79lj\"" Nov 04 04:46:42.943707 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.943684 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 04:46:42.955669 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.955643 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/27a76981-142e-4959-aac0-daaffc4f9b95-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:46:42.955786 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.955672 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-run\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:42.955786 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.955690 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jj28\" (UniqueName: \"kubernetes.io/projected/63e7d159-5f5f-4802-997b-5bfc47f1dad2-kube-api-access-2jj28\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:42.955786 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.955714 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-run-openvswitch\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:42.955786 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.955730 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64661f31-04d2-41e7-aab7-28000a68fc64-ovnkube-config\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:42.955984 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.955786 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-cnibin\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:42.955984 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.955849 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-multus-socket-dir-parent\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:42.955984 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.955884 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-run-k8s-cni-cncf-io\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:42.955984 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.955930 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-var-lib-cni-multus\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:42.955984 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.955964 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24bc4691-e47f-4f98-ad47-f14654d95d28-kubelet-config\") pod \"global-pull-secret-syncer-m5khv\" (UID: \"24bc4691-e47f-4f98-ad47-f14654d95d28\") " pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:46:42.956253 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956005 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-kubelet\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:42.956253 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956044 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-run-ovn-kubernetes\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:42.956253 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956066 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/565b853c-b917-41a5-b739-368db09dac1f-host-slash\") pod \"iptables-alerter-znffl\" (UID: \"565b853c-b917-41a5-b739-368db09dac1f\") " pod="openshift-network-operator/iptables-alerter-znffl" Nov 04 04:46:42.956253 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956082 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-multus-conf-dir\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:42.956253 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956120 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-systemd\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:42.956253 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956149 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-node-log\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:42.956253 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956174 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-log-socket\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:42.956253 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956211 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f1a157-ce5b-4040-98de-ed53507268db-metrics-certs\") pod \"network-metrics-daemon-pbjtk\" (UID: \"12f1a157-ce5b-4040-98de-ed53507268db\") " pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:46:42.956253 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956248 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-etc-openvswitch\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:42.956607 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956289 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-cni-bin\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:42.956607 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956306 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjmwq\" (UniqueName: \"kubernetes.io/projected/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-kube-api-access-jjmwq\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:42.956607 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956334 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24bc4691-e47f-4f98-ad47-f14654d95d28-original-pull-secret\") pod \"global-pull-secret-syncer-m5khv\" (UID: \"24bc4691-e47f-4f98-ad47-f14654d95d28\") " pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:46:42.956607 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:42.956352 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:46:42.956607 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956358 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-systemd-units\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:42.956607 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956385 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/030a8f10-c4b1-45b2-abb8-88ed48e53a80-agent-certs\") pod \"konnectivity-agent-7csxg\" (UID: \"030a8f10-c4b1-45b2-abb8-88ed48e53a80\") " pod="kube-system/konnectivity-agent-7csxg" Nov 04 04:46:42.956607 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956466 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-kubernetes\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:42.956607 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:42.956525 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12f1a157-ce5b-4040-98de-ed53507268db-metrics-certs podName:12f1a157-ce5b-4040-98de-ed53507268db nodeName:}" failed. No retries permitted until 2026-11-04 04:46:43.456490086 +0000 UTC m=+3.041758823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12f1a157-ce5b-4040-98de-ed53507268db-metrics-certs") pod "network-metrics-daemon-pbjtk" (UID: "12f1a157-ce5b-4040-98de-ed53507268db") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:46:42.956607 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956548 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct442\" (UniqueName: \"kubernetes.io/projected/d0e2510f-0579-4594-bd3b-a3dadf3fb81a-kube-api-access-ct442\") pod \"network-check-target-6xc6w\" (UID: \"d0e2510f-0579-4594-bd3b-a3dadf3fb81a\") " pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:46:42.956607 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956566 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-cni-netd\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:42.956607 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956583 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64661f31-04d2-41e7-aab7-28000a68fc64-ovn-node-metrics-cert\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:42.956607 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956602 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27a76981-142e-4959-aac0-daaffc4f9b95-cni-binary-copy\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:46:42.957024 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956617 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-host\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:42.957024 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956632 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed8f18be-cea2-4c8f-a143-25ddd48b1a74-tmp-dir\") pod \"node-resolver-wwxmc\" (UID: \"ed8f18be-cea2-4c8f-a143-25ddd48b1a74\") " pod="openshift-dns/node-resolver-wwxmc" Nov 04 04:46:42.957024 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956678 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-multus-cni-dir\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:42.957024 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956718 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-cni-binary-copy\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:42.957024 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956716 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 04:46:42.957024 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956769 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27a76981-142e-4959-aac0-daaffc4f9b95-cnibin\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:46:42.957024 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956802 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-registration-dir\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:46:42.957024 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956827 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd2j4\" (UniqueName: \"kubernetes.io/projected/26237d52-9119-41f2-919a-5e5153ea223a-kube-api-access-nd2j4\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:46:42.957024 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956880 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-registration-dir\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:46:42.957024 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956889 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-etc-kubernetes\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:42.957024 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956920 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-run-netns\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:42.957024 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.956949 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64661f31-04d2-41e7-aab7-28000a68fc64-ovnkube-script-lib\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:42.957475 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957038 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-system-cni-dir\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:42.957475 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957084 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-socket-dir\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:46:42.957475 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957102 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-etc-selinux\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:46:42.957475 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957117 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-sysctl-d\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:42.957475 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957134 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed8f18be-cea2-4c8f-a143-25ddd48b1a74-hosts-file\") pod \"node-resolver-wwxmc\" (UID: \"ed8f18be-cea2-4c8f-a143-25ddd48b1a74\") " pod="openshift-dns/node-resolver-wwxmc" Nov 04 04:46:42.957475 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957151 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjj9c\" (UniqueName: \"kubernetes.io/projected/acddcc2f-2b31-478b-a0de-552929cd7bd4-kube-api-access-rjj9c\") pod \"node-ca-lfjtg\" (UID: \"acddcc2f-2b31-478b-a0de-552929cd7bd4\") " pod="openshift-image-registry/node-ca-lfjtg" Nov 04 04:46:42.957475 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957180 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-hostroot\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:42.957475 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957217 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27a76981-142e-4959-aac0-daaffc4f9b95-system-cni-dir\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:46:42.957475 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957251 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-etc-selinux\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:46:42.957475 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957287 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxwbx\" (UniqueName: \"kubernetes.io/projected/27a76981-142e-4959-aac0-daaffc4f9b95-kube-api-access-xxwbx\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:46:42.957475 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957327 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-var-lib-openvswitch\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:42.957475 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957334 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-socket-dir\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:46:42.957475 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957355 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/565b853c-b917-41a5-b739-368db09dac1f-iptables-alerter-script\") pod \"iptables-alerter-znffl\" (UID: \"565b853c-b917-41a5-b739-368db09dac1f\") " pod="openshift-network-operator/iptables-alerter-znffl" Nov 04 04:46:42.957475 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957402 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-var-lib-cni-bin\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:42.957475 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957464 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-run-multus-certs\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:42.958147 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957488 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-sysctl-conf\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:42.958147 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957510 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-var-lib-kubelet\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:42.958147 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957531 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63e7d159-5f5f-4802-997b-5bfc47f1dad2-tmp\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:42.958147 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957560 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acddcc2f-2b31-478b-a0de-552929cd7bd4-host\") pod \"node-ca-lfjtg\" (UID: \"acddcc2f-2b31-478b-a0de-552929cd7bd4\") " pod="openshift-image-registry/node-ca-lfjtg" Nov 04 04:46:42.958147 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957589 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-device-dir\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:46:42.958147 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957645 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-device-dir\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:46:42.958147 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957648 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-sys-fs\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:46:42.958147 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957692 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-modprobe-d\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:42.958147 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957707 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-lib-modules\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:42.958147 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957760 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-sys-fs\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:46:42.958147 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957803 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w87hs\" (UniqueName: \"kubernetes.io/projected/ed8f18be-cea2-4c8f-a143-25ddd48b1a74-kube-api-access-w87hs\") pod \"node-resolver-wwxmc\" (UID: \"ed8f18be-cea2-4c8f-a143-25ddd48b1a74\") " pod="openshift-dns/node-resolver-wwxmc" Nov 04 04:46:42.958147 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957831 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8q7s\" (UniqueName: \"kubernetes.io/projected/64661f31-04d2-41e7-aab7-28000a68fc64-kube-api-access-q8q7s\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:42.958147 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957856 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-sys\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:42.958147 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957922 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-slash\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:42.958147 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957950 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-run-systemd\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:42.958147 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.957975 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-run-ovn\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:42.958147 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.958001 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9g5g\" (UniqueName: \"kubernetes.io/projected/565b853c-b917-41a5-b739-368db09dac1f-kube-api-access-m9g5g\") pod \"iptables-alerter-znffl\" (UID: \"565b853c-b917-41a5-b739-368db09dac1f\") " pod="openshift-network-operator/iptables-alerter-znffl" Nov 04 04:46:42.958912 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.958046 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-run-netns\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:42.958912 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.958086 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-multus-daemon-config\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:42.958912 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.958122 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27a76981-142e-4959-aac0-daaffc4f9b95-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:46:42.958912 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.958171 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/030a8f10-c4b1-45b2-abb8-88ed48e53a80-konnectivity-ca\") pod \"konnectivity-agent-7csxg\" (UID: \"030a8f10-c4b1-45b2-abb8-88ed48e53a80\") " pod="kube-system/konnectivity-agent-7csxg" Nov 04 04:46:42.958912 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.958201 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-sysconfig\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:42.958912 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.958254 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:42.958912 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.958316 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-var-lib-kubelet\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:42.958912 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.958366 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98rdv\" (UniqueName: \"kubernetes.io/projected/12f1a157-ce5b-4040-98de-ed53507268db-kube-api-access-98rdv\") pod \"network-metrics-daemon-pbjtk\" (UID: \"12f1a157-ce5b-4040-98de-ed53507268db\") " pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:46:42.958912 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.958395 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:46:42.958912 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.958423 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24bc4691-e47f-4f98-ad47-f14654d95d28-dbus\") pod \"global-pull-secret-syncer-m5khv\" (UID: \"24bc4691-e47f-4f98-ad47-f14654d95d28\") " pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:46:42.958912 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.958452 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-tuned\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:42.958912 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.958478 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64661f31-04d2-41e7-aab7-28000a68fc64-env-overrides\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:42.958912 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.958485 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:46:42.958912 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.958502 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acddcc2f-2b31-478b-a0de-552929cd7bd4-serviceca\") pod \"node-ca-lfjtg\" (UID: \"acddcc2f-2b31-478b-a0de-552929cd7bd4\") " pod="openshift-image-registry/node-ca-lfjtg" Nov 04 04:46:42.958912 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.958529 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/27a76981-142e-4959-aac0-daaffc4f9b95-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:46:42.958912 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.958903 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/030a8f10-c4b1-45b2-abb8-88ed48e53a80-konnectivity-ca\") pod \"konnectivity-agent-7csxg\" (UID: \"030a8f10-c4b1-45b2-abb8-88ed48e53a80\") " pod="kube-system/konnectivity-agent-7csxg" Nov 04 04:46:42.961311 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.961274 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/030a8f10-c4b1-45b2-abb8-88ed48e53a80-agent-certs\") pod \"konnectivity-agent-7csxg\" (UID: \"030a8f10-c4b1-45b2-abb8-88ed48e53a80\") " pod="kube-system/konnectivity-agent-7csxg" Nov 04 04:46:42.999755 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.999701 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 04:46:42.999877 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:42.999770 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sdltl" Nov 04 04:46:43.005585 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.005556 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 04:46:43.005944 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.005917 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 04:46:43.006063 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.005949 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 04:46:43.006063 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.005966 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-shz8v\"" Nov 04 04:46:43.006063 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.005949 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 04:46:43.006063 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.006057 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 04:46:43.006250 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.006169 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 04:46:43.006250 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.006199 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mkkxt\"" Nov 04 04:46:43.006345 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.006256 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 04:46:43.006345 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.006257 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ssxrh\"" Nov 04 04:46:43.006345 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.006262 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zsr9v\"" Nov 04 04:46:43.006345 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.006278 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 04:46:43.006345 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.006278 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kgq44\"" Nov 04 04:46:43.006345 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.006300 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 04:46:43.006345 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.006330 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 04:46:43.006345 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.006339 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 04:46:43.006618 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.006397 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 04:46:43.006618 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.006448 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 04:46:43.006618 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.006465 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 04:46:43.006618 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.006489 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 04:46:43.006618 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.006500 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 04:46:43.015737 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.015722 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 04:46:43.015737 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.015736 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 04:46:43.015842 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.015726 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 04:46:43.016097 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.016085 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vmp7s\"" Nov 04 04:46:43.016154 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.016144 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 04:46:43.016154 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.016150 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tqbjc\"" Nov 04 04:46:43.016258 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.016203 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 04:46:43.059108 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059074 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24bc4691-e47f-4f98-ad47-f14654d95d28-dbus\") pod \"global-pull-secret-syncer-m5khv\" (UID: \"24bc4691-e47f-4f98-ad47-f14654d95d28\") " pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:46:43.059108 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059107 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-tuned\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.059242 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059126 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64661f31-04d2-41e7-aab7-28000a68fc64-env-overrides\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.059242 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059150 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acddcc2f-2b31-478b-a0de-552929cd7bd4-serviceca\") pod \"node-ca-lfjtg\" (UID: \"acddcc2f-2b31-478b-a0de-552929cd7bd4\") " pod="openshift-image-registry/node-ca-lfjtg" Nov 04 04:46:43.059242 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059190 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/27a76981-142e-4959-aac0-daaffc4f9b95-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:46:43.059242 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059218 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/27a76981-142e-4959-aac0-daaffc4f9b95-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:46:43.059403 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059256 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-run\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.059403 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059286 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jj28\" (UniqueName: \"kubernetes.io/projected/63e7d159-5f5f-4802-997b-5bfc47f1dad2-kube-api-access-2jj28\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.059403 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059310 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-run-openvswitch\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.059403 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059335 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64661f31-04d2-41e7-aab7-28000a68fc64-ovnkube-config\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.059403 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059360 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-cnibin\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.059403 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059383 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-multus-socket-dir-parent\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.059717 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059408 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-run-k8s-cni-cncf-io\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.059717 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059432 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-var-lib-cni-multus\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.059717 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059425 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-run-openvswitch\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.059717 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059467 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24bc4691-e47f-4f98-ad47-f14654d95d28-kubelet-config\") pod \"global-pull-secret-syncer-m5khv\" (UID: \"24bc4691-e47f-4f98-ad47-f14654d95d28\") " pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:46:43.059717 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059472 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24bc4691-e47f-4f98-ad47-f14654d95d28-dbus\") pod \"global-pull-secret-syncer-m5khv\" (UID: \"24bc4691-e47f-4f98-ad47-f14654d95d28\") " pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:46:43.059717 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059492 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-kubelet\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.059717 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059517 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-run-ovn-kubernetes\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.059717 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059532 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-var-lib-cni-multus\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.059717 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059540 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/565b853c-b917-41a5-b739-368db09dac1f-host-slash\") pod \"iptables-alerter-znffl\" (UID: \"565b853c-b917-41a5-b739-368db09dac1f\") " pod="openshift-network-operator/iptables-alerter-znffl" Nov 04 04:46:43.059717 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059542 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-multus-socket-dir-parent\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.059717 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059565 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-multus-conf-dir\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.059717 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059583 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-run-k8s-cni-cncf-io\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.059717 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059592 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-systemd\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.059717 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059615 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-node-log\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.059717 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059633 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-kubelet\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.059717 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059430 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-run\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.059717 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-log-socket\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.059717 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059694 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24bc4691-e47f-4f98-ad47-f14654d95d28-kubelet-config\") pod \"global-pull-secret-syncer-m5khv\" (UID: \"24bc4691-e47f-4f98-ad47-f14654d95d28\") " pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:46:43.060697 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059699 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-etc-openvswitch\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.060697 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059736 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-cni-bin\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.060697 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059748 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-etc-openvswitch\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.060697 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjmwq\" (UniqueName: \"kubernetes.io/projected/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-kube-api-access-jjmwq\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.060697 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059794 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24bc4691-e47f-4f98-ad47-f14654d95d28-original-pull-secret\") pod \"global-pull-secret-syncer-m5khv\" (UID: \"24bc4691-e47f-4f98-ad47-f14654d95d28\") " pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:46:43.060697 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059802 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-run-ovn-kubernetes\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.060697 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059819 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-systemd-units\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.060697 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059851 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/565b853c-b917-41a5-b739-368db09dac1f-host-slash\") pod \"iptables-alerter-znffl\" (UID: \"565b853c-b917-41a5-b739-368db09dac1f\") " pod="openshift-network-operator/iptables-alerter-znffl" Nov 04 04:46:43.060697 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059853 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-kubernetes\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.060697 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059893 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-kubernetes\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.060697 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059893 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/27a76981-142e-4959-aac0-daaffc4f9b95-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:46:43.060697 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059895 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct442\" (UniqueName: \"kubernetes.io/projected/d0e2510f-0579-4594-bd3b-a3dadf3fb81a-kube-api-access-ct442\") pod \"network-check-target-6xc6w\" (UID: \"d0e2510f-0579-4594-bd3b-a3dadf3fb81a\") " pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:46:43.060697 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059946 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-cni-netd\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.060697 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:43.059972 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 04:46:43.060697 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:43.060040 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24bc4691-e47f-4f98-ad47-f14654d95d28-original-pull-secret podName:24bc4691-e47f-4f98-ad47-f14654d95d28 nodeName:}" failed. No retries permitted until 2026-11-04 04:46:43.560021068 +0000 UTC m=+3.145289785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24bc4691-e47f-4f98-ad47-f14654d95d28-original-pull-secret") pod "global-pull-secret-syncer-m5khv" (UID: "24bc4691-e47f-4f98-ad47-f14654d95d28") : object "kube-system"/"original-pull-secret" not registered Nov 04 04:46:43.060697 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060109 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-cni-bin\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.060697 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060150 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 04:46:43.060697 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060167 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-cnibin\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.061629 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059975 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64661f31-04d2-41e7-aab7-28000a68fc64-ovn-node-metrics-cert\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.061629 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060307 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-multus-conf-dir\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.061629 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060317 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27a76981-142e-4959-aac0-daaffc4f9b95-cni-binary-copy\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:46:43.061629 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060370 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-host\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.061629 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060383 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-systemd\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.061629 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060395 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed8f18be-cea2-4c8f-a143-25ddd48b1a74-tmp-dir\") pod \"node-resolver-wwxmc\" (UID: \"ed8f18be-cea2-4c8f-a143-25ddd48b1a74\") " pod="openshift-dns/node-resolver-wwxmc" Nov 04 04:46:43.061629 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060456 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-node-log\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.061629 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060494 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-multus-cni-dir\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.061629 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060529 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-cni-binary-copy\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.061629 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060557 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27a76981-142e-4959-aac0-daaffc4f9b95-cnibin\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:46:43.061629 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060587 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-etc-kubernetes\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.061629 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060591 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acddcc2f-2b31-478b-a0de-552929cd7bd4-serviceca\") pod \"node-ca-lfjtg\" (UID: \"acddcc2f-2b31-478b-a0de-552929cd7bd4\") " pod="openshift-image-registry/node-ca-lfjtg" Nov 04 04:46:43.061629 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060616 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-run-netns\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.061629 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060643 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64661f31-04d2-41e7-aab7-28000a68fc64-ovnkube-script-lib\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.061629 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060670 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-system-cni-dir\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.061629 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060698 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-sysctl-d\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.061629 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060721 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed8f18be-cea2-4c8f-a143-25ddd48b1a74-hosts-file\") pod \"node-resolver-wwxmc\" (UID: \"ed8f18be-cea2-4c8f-a143-25ddd48b1a74\") " pod="openshift-dns/node-resolver-wwxmc" Nov 04 04:46:43.061629 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060747 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjj9c\" (UniqueName: \"kubernetes.io/projected/acddcc2f-2b31-478b-a0de-552929cd7bd4-kube-api-access-rjj9c\") pod \"node-ca-lfjtg\" (UID: \"acddcc2f-2b31-478b-a0de-552929cd7bd4\") " pod="openshift-image-registry/node-ca-lfjtg" Nov 04 04:46:43.062502 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060752 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed8f18be-cea2-4c8f-a143-25ddd48b1a74-tmp-dir\") pod \"node-resolver-wwxmc\" (UID: \"ed8f18be-cea2-4c8f-a143-25ddd48b1a74\") " pod="openshift-dns/node-resolver-wwxmc" Nov 04 04:46:43.062502 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060774 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-hostroot\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.062502 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060691 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-systemd-units\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.062502 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060803 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27a76981-142e-4959-aac0-daaffc4f9b95-system-cni-dir\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:46:43.062502 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060855 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27a76981-142e-4959-aac0-daaffc4f9b95-system-cni-dir\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:46:43.062502 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060867 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-run-netns\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.062502 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060921 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-log-socket\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.062502 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.060923 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-system-cni-dir\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.062502 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.059854 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/27a76981-142e-4959-aac0-daaffc4f9b95-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:46:43.062502 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.061127 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64661f31-04d2-41e7-aab7-28000a68fc64-env-overrides\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.062502 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.061188 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-multus-cni-dir\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.062502 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.061444 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-cni-netd\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.062502 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.061572 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed8f18be-cea2-4c8f-a143-25ddd48b1a74-hosts-file\") pod \"node-resolver-wwxmc\" (UID: \"ed8f18be-cea2-4c8f-a143-25ddd48b1a74\") " pod="openshift-dns/node-resolver-wwxmc" Nov 04 04:46:43.062502 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.061681 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-host\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.062502 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.061709 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64661f31-04d2-41e7-aab7-28000a68fc64-ovnkube-config\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.062502 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.061716 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-sysctl-d\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.062502 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.061783 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-hostroot\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.062502 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.061783 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-etc-kubernetes\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.063332 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.061816 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27a76981-142e-4959-aac0-daaffc4f9b95-cnibin\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:46:43.063332 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.061818 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-cni-binary-copy\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.063332 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.061828 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxwbx\" (UniqueName: \"kubernetes.io/projected/27a76981-142e-4959-aac0-daaffc4f9b95-kube-api-access-xxwbx\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:46:43.063332 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.061868 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-var-lib-openvswitch\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.063332 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.061897 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/565b853c-b917-41a5-b739-368db09dac1f-iptables-alerter-script\") pod \"iptables-alerter-znffl\" (UID: \"565b853c-b917-41a5-b739-368db09dac1f\") " pod="openshift-network-operator/iptables-alerter-znffl" Nov 04 04:46:43.063332 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.061932 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-var-lib-cni-bin\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.063332 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.061937 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-var-lib-openvswitch\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.063332 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.061956 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-run-multus-certs\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.063332 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062038 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-sysctl-conf\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.063332 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062046 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-var-lib-cni-bin\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.063332 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062092 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-run-multus-certs\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.063332 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062142 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-var-lib-kubelet\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.063332 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062212 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63e7d159-5f5f-4802-997b-5bfc47f1dad2-tmp\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.063332 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062236 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-sysctl-conf\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.063332 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062265 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-var-lib-kubelet\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.063332 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062297 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acddcc2f-2b31-478b-a0de-552929cd7bd4-host\") pod \"node-ca-lfjtg\" (UID: \"acddcc2f-2b31-478b-a0de-552929cd7bd4\") " pod="openshift-image-registry/node-ca-lfjtg" Nov 04 04:46:43.063332 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062347 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acddcc2f-2b31-478b-a0de-552929cd7bd4-host\") pod \"node-ca-lfjtg\" (UID: \"acddcc2f-2b31-478b-a0de-552929cd7bd4\") " pod="openshift-image-registry/node-ca-lfjtg" Nov 04 04:46:43.063332 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062352 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-modprobe-d\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.063902 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062377 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-lib-modules\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.063902 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062394 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w87hs\" (UniqueName: \"kubernetes.io/projected/ed8f18be-cea2-4c8f-a143-25ddd48b1a74-kube-api-access-w87hs\") pod \"node-resolver-wwxmc\" (UID: \"ed8f18be-cea2-4c8f-a143-25ddd48b1a74\") " pod="openshift-dns/node-resolver-wwxmc" Nov 04 04:46:43.063902 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062409 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8q7s\" (UniqueName: \"kubernetes.io/projected/64661f31-04d2-41e7-aab7-28000a68fc64-kube-api-access-q8q7s\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.063902 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062427 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-sys\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.063902 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062464 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-slash\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.063902 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062482 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-run-systemd\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.063902 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062498 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-run-ovn\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.063902 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062508 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-modprobe-d\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.063902 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062520 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9g5g\" (UniqueName: \"kubernetes.io/projected/565b853c-b917-41a5-b739-368db09dac1f-kube-api-access-m9g5g\") pod \"iptables-alerter-znffl\" (UID: \"565b853c-b917-41a5-b739-368db09dac1f\") " pod="openshift-network-operator/iptables-alerter-znffl" Nov 04 04:46:43.063902 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062547 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-run-netns\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.063902 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062553 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-run-systemd\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.063902 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062559 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-run-ovn\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.063902 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062594 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-multus-daemon-config\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.063902 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062603 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-run-netns\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.063902 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062629 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27a76981-142e-4959-aac0-daaffc4f9b95-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:46:43.063902 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062657 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-sysconfig\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.063902 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062606 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-slash\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.064459 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062685 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.064459 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062713 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-var-lib-kubelet\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.064459 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062719 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-sys\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.064459 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062816 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-lib-modules\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.064459 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062948 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27a76981-142e-4959-aac0-daaffc4f9b95-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:46:43.064459 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.062992 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27a76981-142e-4959-aac0-daaffc4f9b95-cni-binary-copy\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:46:43.064459 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.063003 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-sysconfig\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.064459 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.063042 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.064459 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.063076 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64661f31-04d2-41e7-aab7-28000a68fc64-ovn-node-metrics-cert\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.064459 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.063108 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-var-lib-kubelet\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.064459 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.063116 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-tuned\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.064459 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.063245 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-multus-daemon-config\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.064459 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.063353 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/565b853c-b917-41a5-b739-368db09dac1f-iptables-alerter-script\") pod \"iptables-alerter-znffl\" (UID: \"565b853c-b917-41a5-b739-368db09dac1f\") " pod="openshift-network-operator/iptables-alerter-znffl" Nov 04 04:46:43.064459 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.063525 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64661f31-04d2-41e7-aab7-28000a68fc64-ovnkube-script-lib\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.064459 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.063525 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd2j4\" (UniqueName: \"kubernetes.io/projected/26237d52-9119-41f2-919a-5e5153ea223a-kube-api-access-nd2j4\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:46:43.065155 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.065134 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98rdv\" (UniqueName: \"kubernetes.io/projected/12f1a157-ce5b-4040-98de-ed53507268db-kube-api-access-98rdv\") pod \"network-metrics-daemon-pbjtk\" (UID: \"12f1a157-ce5b-4040-98de-ed53507268db\") " pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:46:43.065476 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.065460 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63e7d159-5f5f-4802-997b-5bfc47f1dad2-tmp\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.119557 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:43.119533 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 04:46:43.119557 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:43.119555 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 04:46:43.119658 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:43.119565 2577 projected.go:196] Error preparing data for projected volume kube-api-access-ct442 for pod openshift-network-diagnostics/network-check-target-6xc6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:46:43.119658 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:43.119612 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0e2510f-0579-4594-bd3b-a3dadf3fb81a-kube-api-access-ct442 podName:d0e2510f-0579-4594-bd3b-a3dadf3fb81a nodeName:}" failed. No retries permitted until 2026-11-04 04:46:43.61959827 +0000 UTC m=+3.204867001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ct442" (UniqueName: "kubernetes.io/projected/d0e2510f-0579-4594-bd3b-a3dadf3fb81a-kube-api-access-ct442") pod "network-check-target-6xc6w" (UID: "d0e2510f-0579-4594-bd3b-a3dadf3fb81a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:46:43.122393 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.122364 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjmwq\" (UniqueName: \"kubernetes.io/projected/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-kube-api-access-jjmwq\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.123053 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.123029 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jj28\" (UniqueName: \"kubernetes.io/projected/63e7d159-5f5f-4802-997b-5bfc47f1dad2-kube-api-access-2jj28\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.123360 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.123344 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjj9c\" (UniqueName: \"kubernetes.io/projected/acddcc2f-2b31-478b-a0de-552929cd7bd4-kube-api-access-rjj9c\") pod \"node-ca-lfjtg\" (UID: \"acddcc2f-2b31-478b-a0de-552929cd7bd4\") " pod="openshift-image-registry/node-ca-lfjtg" Nov 04 04:46:43.123642 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.123621 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9g5g\" (UniqueName: \"kubernetes.io/projected/565b853c-b917-41a5-b739-368db09dac1f-kube-api-access-m9g5g\") pod \"iptables-alerter-znffl\" (UID: \"565b853c-b917-41a5-b739-368db09dac1f\") " pod="openshift-network-operator/iptables-alerter-znffl" Nov 04 04:46:43.124129 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.124113 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxwbx\" (UniqueName: \"kubernetes.io/projected/27a76981-142e-4959-aac0-daaffc4f9b95-kube-api-access-xxwbx\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:46:43.124486 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.124472 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w87hs\" (UniqueName: \"kubernetes.io/projected/ed8f18be-cea2-4c8f-a143-25ddd48b1a74-kube-api-access-w87hs\") pod \"node-resolver-wwxmc\" (UID: \"ed8f18be-cea2-4c8f-a143-25ddd48b1a74\") " pod="openshift-dns/node-resolver-wwxmc" Nov 04 04:46:43.124625 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.124609 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8q7s\" (UniqueName: \"kubernetes.io/projected/64661f31-04d2-41e7-aab7-28000a68fc64-kube-api-access-q8q7s\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.152485 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.152464 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7csxg" Nov 04 04:46:43.162929 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:43.162906 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030a8f10_c4b1_45b2_abb8_88ed48e53a80.slice/crio-c9abbebf0d2223426b71b38556e7fd55a7058957a43cc318c0e6309ba72d5c46 WatchSource:0}: Error finding container c9abbebf0d2223426b71b38556e7fd55a7058957a43cc318c0e6309ba72d5c46: Status 404 returned error can't find the container with id c9abbebf0d2223426b71b38556e7fd55a7058957a43cc318c0e6309ba72d5c46 Nov 04 04:46:43.163531 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.163512 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:46:43.167726 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.167206 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:46:43.170832 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.170815 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lfjtg" Nov 04 04:46:43.174906 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:43.174881 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26237d52_9119_41f2_919a_5e5153ea223a.slice/crio-cd738a1ec5d9f08c2efc725dd095c882117ba37f285517c37650b400964c93d4 WatchSource:0}: Error finding container cd738a1ec5d9f08c2efc725dd095c882117ba37f285517c37650b400964c93d4: Status 404 returned error can't find the container with id cd738a1ec5d9f08c2efc725dd095c882117ba37f285517c37650b400964c93d4 Nov 04 04:46:43.175538 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:43.175518 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e7d159_5f5f_4802_997b_5bfc47f1dad2.slice/crio-f0f5a6e8636d53e263123feed3ce7542fe355a8ba44a0c4d8d6ba21c8041cbf9 WatchSource:0}: Error finding container f0f5a6e8636d53e263123feed3ce7542fe355a8ba44a0c4d8d6ba21c8041cbf9: Status 404 returned error can't find the container with id f0f5a6e8636d53e263123feed3ce7542fe355a8ba44a0c4d8d6ba21c8041cbf9 Nov 04 04:46:43.177061 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.177041 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-znffl" Nov 04 04:46:43.180988 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:43.180971 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacddcc2f_2b31_478b_a0de_552929cd7bd4.slice/crio-c60f1c25de9e2eb62b826c1324da16737ba57c99e55c4ed0ff2044571e318911 WatchSource:0}: Error finding container c60f1c25de9e2eb62b826c1324da16737ba57c99e55c4ed0ff2044571e318911: Status 404 returned error can't find the container with id c60f1c25de9e2eb62b826c1324da16737ba57c99e55c4ed0ff2044571e318911 Nov 04 04:46:43.183123 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.183102 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wwxmc" Nov 04 04:46:43.186367 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:43.186349 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565b853c_b917_41a5_b739_368db09dac1f.slice/crio-fb0861f4193124e2573bd818c65ccf7f2bb0bd9e1a74049ad4341c293ad48882 WatchSource:0}: Error finding container fb0861f4193124e2573bd818c65ccf7f2bb0bd9e1a74049ad4341c293ad48882: Status 404 returned error can't find the container with id fb0861f4193124e2573bd818c65ccf7f2bb0bd9e1a74049ad4341c293ad48882 Nov 04 04:46:43.188408 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.188392 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cxp8f" Nov 04 04:46:43.192800 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:43.192778 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8f18be_cea2_4c8f_a143_25ddd48b1a74.slice/crio-9fba49fabd21e22b7aa117c7a67b47816b1f2967db50873193f3b1b6b35e4fc3 WatchSource:0}: Error finding container 9fba49fabd21e22b7aa117c7a67b47816b1f2967db50873193f3b1b6b35e4fc3: Status 404 returned error can't find the container with id 9fba49fabd21e22b7aa117c7a67b47816b1f2967db50873193f3b1b6b35e4fc3 Nov 04 04:46:43.197479 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:43.197460 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9a4dfc_3b04_4a71_ab1c_dd5da4b22c8b.slice/crio-4d83e82749c5a46c027df325004fd629f95cf83c9a4b282a3fe605c67574425f WatchSource:0}: Error finding container 4d83e82749c5a46c027df325004fd629f95cf83c9a4b282a3fe605c67574425f: Status 404 returned error can't find the container with id 4d83e82749c5a46c027df325004fd629f95cf83c9a4b282a3fe605c67574425f Nov 04 04:46:43.218656 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.218638 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:46:43.224893 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.224864 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:46:43.228584 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:43.228565 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64661f31_04d2_41e7_aab7_28000a68fc64.slice/crio-4b783f131c1ac45c5c286605c31b12d834ba7248aed8d4c244a1e791fdd5beff WatchSource:0}: Error finding container 4b783f131c1ac45c5c286605c31b12d834ba7248aed8d4c244a1e791fdd5beff: Status 404 returned error can't find the container with id 4b783f131c1ac45c5c286605c31b12d834ba7248aed8d4c244a1e791fdd5beff Nov 04 04:46:43.234930 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:46:43.234908 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a76981_142e_4959_aac0_daaffc4f9b95.slice/crio-c4b7c4e9635fe7d938e45641be9f043fd487a93b37ebac50bacca07a81b1ba51 WatchSource:0}: Error finding container c4b7c4e9635fe7d938e45641be9f043fd487a93b37ebac50bacca07a81b1ba51: Status 404 returned error can't find the container with id c4b7c4e9635fe7d938e45641be9f043fd487a93b37ebac50bacca07a81b1ba51 Nov 04 04:46:43.464903 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.464816 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f1a157-ce5b-4040-98de-ed53507268db-metrics-certs\") pod \"network-metrics-daemon-pbjtk\" (UID: \"12f1a157-ce5b-4040-98de-ed53507268db\") " pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:46:43.465072 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:43.464975 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:46:43.465072 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:43.465052 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12f1a157-ce5b-4040-98de-ed53507268db-metrics-certs podName:12f1a157-ce5b-4040-98de-ed53507268db nodeName:}" failed. No retries permitted until 2026-11-04 04:46:44.465034104 +0000 UTC m=+4.050302818 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12f1a157-ce5b-4040-98de-ed53507268db-metrics-certs") pod "network-metrics-daemon-pbjtk" (UID: "12f1a157-ce5b-4040-98de-ed53507268db") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:46:43.566370 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.566324 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24bc4691-e47f-4f98-ad47-f14654d95d28-original-pull-secret\") pod \"global-pull-secret-syncer-m5khv\" (UID: \"24bc4691-e47f-4f98-ad47-f14654d95d28\") " pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:46:43.566562 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:43.566485 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 04:46:43.566562 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:43.566553 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24bc4691-e47f-4f98-ad47-f14654d95d28-original-pull-secret podName:24bc4691-e47f-4f98-ad47-f14654d95d28 nodeName:}" failed. No retries permitted until 2026-11-04 04:46:44.566534092 +0000 UTC m=+4.151802810 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24bc4691-e47f-4f98-ad47-f14654d95d28-original-pull-secret") pod "global-pull-secret-syncer-m5khv" (UID: "24bc4691-e47f-4f98-ad47-f14654d95d28") : object "kube-system"/"original-pull-secret" not registered Nov 04 04:46:43.667508 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.667476 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct442\" (UniqueName: \"kubernetes.io/projected/d0e2510f-0579-4594-bd3b-a3dadf3fb81a-kube-api-access-ct442\") pod \"network-check-target-6xc6w\" (UID: \"d0e2510f-0579-4594-bd3b-a3dadf3fb81a\") " pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:46:43.667706 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:43.667625 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 04:46:43.667706 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:43.667648 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 04:46:43.667706 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:43.667662 2577 projected.go:196] Error preparing data for projected volume kube-api-access-ct442 for pod openshift-network-diagnostics/network-check-target-6xc6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:46:43.667835 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:43.667712 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0e2510f-0579-4594-bd3b-a3dadf3fb81a-kube-api-access-ct442 podName:d0e2510f-0579-4594-bd3b-a3dadf3fb81a nodeName:}" failed. No retries permitted until 2026-11-04 04:46:44.667693685 +0000 UTC m=+4.252962619 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ct442" (UniqueName: "kubernetes.io/projected/d0e2510f-0579-4594-bd3b-a3dadf3fb81a-kube-api-access-ct442") pod "network-check-target-6xc6w" (UID: "d0e2510f-0579-4594-bd3b-a3dadf3fb81a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:46:43.829123 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.829027 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wwxmc" event={"ID":"ed8f18be-cea2-4c8f-a143-25ddd48b1a74","Type":"ContainerStarted","Data":"9fba49fabd21e22b7aa117c7a67b47816b1f2967db50873193f3b1b6b35e4fc3"} Nov 04 04:46:43.830403 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.830376 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-znffl" event={"ID":"565b853c-b917-41a5-b739-368db09dac1f","Type":"ContainerStarted","Data":"fb0861f4193124e2573bd818c65ccf7f2bb0bd9e1a74049ad4341c293ad48882"} Nov 04 04:46:43.831839 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.831809 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lfjtg" event={"ID":"acddcc2f-2b31-478b-a0de-552929cd7bd4","Type":"ContainerStarted","Data":"c60f1c25de9e2eb62b826c1324da16737ba57c99e55c4ed0ff2044571e318911"} Nov 04 04:46:43.833237 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.833210 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7csxg" event={"ID":"030a8f10-c4b1-45b2-abb8-88ed48e53a80","Type":"ContainerStarted","Data":"c9abbebf0d2223426b71b38556e7fd55a7058957a43cc318c0e6309ba72d5c46"} Nov 04 04:46:43.834569 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.834533 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" event={"ID":"64661f31-04d2-41e7-aab7-28000a68fc64","Type":"ContainerStarted","Data":"4b783f131c1ac45c5c286605c31b12d834ba7248aed8d4c244a1e791fdd5beff"} Nov 04 04:46:43.836015 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.835979 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cxp8f" event={"ID":"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b","Type":"ContainerStarted","Data":"4d83e82749c5a46c027df325004fd629f95cf83c9a4b282a3fe605c67574425f"} Nov 04 04:46:43.837752 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.837728 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hlw45" event={"ID":"63e7d159-5f5f-4802-997b-5bfc47f1dad2","Type":"ContainerStarted","Data":"f0f5a6e8636d53e263123feed3ce7542fe355a8ba44a0c4d8d6ba21c8041cbf9"} Nov 04 04:46:43.839452 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.839393 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" event={"ID":"26237d52-9119-41f2-919a-5e5153ea223a","Type":"ContainerStarted","Data":"cd738a1ec5d9f08c2efc725dd095c882117ba37f285517c37650b400964c93d4"} Nov 04 04:46:43.841513 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:43.841488 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rkbf" event={"ID":"27a76981-142e-4959-aac0-daaffc4f9b95","Type":"ContainerStarted","Data":"c4b7c4e9635fe7d938e45641be9f043fd487a93b37ebac50bacca07a81b1ba51"} Nov 04 04:46:44.003396 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:44.003333 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 04:41:43 +0000 UTC" deadline="2028-07-28 04:57:25.472571476 +0000 UTC" Nov 04 04:46:44.003396 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:44.003387 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15168h10m41.469189086s" Nov 04 04:46:44.473581 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:44.473554 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f1a157-ce5b-4040-98de-ed53507268db-metrics-certs\") pod \"network-metrics-daemon-pbjtk\" (UID: \"12f1a157-ce5b-4040-98de-ed53507268db\") " pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:46:44.473719 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:44.473706 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:46:44.473768 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:44.473757 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12f1a157-ce5b-4040-98de-ed53507268db-metrics-certs podName:12f1a157-ce5b-4040-98de-ed53507268db nodeName:}" failed. No retries permitted until 2026-11-04 04:46:46.473744168 +0000 UTC m=+6.059012897 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12f1a157-ce5b-4040-98de-ed53507268db-metrics-certs") pod "network-metrics-daemon-pbjtk" (UID: "12f1a157-ce5b-4040-98de-ed53507268db") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:46:44.574764 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:44.574724 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24bc4691-e47f-4f98-ad47-f14654d95d28-original-pull-secret\") pod \"global-pull-secret-syncer-m5khv\" (UID: \"24bc4691-e47f-4f98-ad47-f14654d95d28\") " pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:46:44.574986 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:44.574861 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 04:46:44.574986 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:44.574925 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24bc4691-e47f-4f98-ad47-f14654d95d28-original-pull-secret podName:24bc4691-e47f-4f98-ad47-f14654d95d28 nodeName:}" failed. No retries permitted until 2026-11-04 04:46:46.574906013 +0000 UTC m=+6.160174753 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24bc4691-e47f-4f98-ad47-f14654d95d28-original-pull-secret") pod "global-pull-secret-syncer-m5khv" (UID: "24bc4691-e47f-4f98-ad47-f14654d95d28") : object "kube-system"/"original-pull-secret" not registered Nov 04 04:46:44.676135 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:44.676094 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct442\" (UniqueName: \"kubernetes.io/projected/d0e2510f-0579-4594-bd3b-a3dadf3fb81a-kube-api-access-ct442\") pod \"network-check-target-6xc6w\" (UID: \"d0e2510f-0579-4594-bd3b-a3dadf3fb81a\") " pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:46:44.676323 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:44.676273 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 04:46:44.676323 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:44.676297 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 04:46:44.676323 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:44.676310 2577 projected.go:196] Error preparing data for projected volume kube-api-access-ct442 for pod openshift-network-diagnostics/network-check-target-6xc6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:46:44.676498 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:44.676372 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0e2510f-0579-4594-bd3b-a3dadf3fb81a-kube-api-access-ct442 podName:d0e2510f-0579-4594-bd3b-a3dadf3fb81a nodeName:}" failed. No retries permitted until 2026-11-04 04:46:46.67635118 +0000 UTC m=+6.261620010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ct442" (UniqueName: "kubernetes.io/projected/d0e2510f-0579-4594-bd3b-a3dadf3fb81a-kube-api-access-ct442") pod "network-check-target-6xc6w" (UID: "d0e2510f-0579-4594-bd3b-a3dadf3fb81a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:46:44.823736 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:44.823701 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:46:44.823889 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:44.823845 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xc6w" podUID="d0e2510f-0579-4594-bd3b-a3dadf3fb81a" Nov 04 04:46:44.823970 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:44.823936 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:46:44.824055 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:44.824019 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbjtk" podUID="12f1a157-ce5b-4040-98de-ed53507268db" Nov 04 04:46:44.824133 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:44.824105 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:46:44.824222 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:44.824179 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5khv" podUID="24bc4691-e47f-4f98-ad47-f14654d95d28" Nov 04 04:46:45.003942 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:45.003855 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 04:41:43 +0000 UTC" deadline="2028-07-07 03:10:50.5278738 +0000 UTC" Nov 04 04:46:45.003942 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:45.003894 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14662h24m5.523984192s" Nov 04 04:46:45.862357 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:45.861504 2577 generic.go:356] "Generic (PLEG): container finished" podID="81b6936b6f0cd5f6748dab38c3bd34d9" containerID="e67fa1f4623c936d0c7325002ca2be90cb66c7270745b0110709670f655a1688" exitCode=0 Nov 04 04:46:45.862357 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:45.861790 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" event={"ID":"81b6936b6f0cd5f6748dab38c3bd34d9","Type":"ContainerDied","Data":"e67fa1f4623c936d0c7325002ca2be90cb66c7270745b0110709670f655a1688"} Nov 04 04:46:46.496368 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:46.496024 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f1a157-ce5b-4040-98de-ed53507268db-metrics-certs\") pod \"network-metrics-daemon-pbjtk\" (UID: \"12f1a157-ce5b-4040-98de-ed53507268db\") " pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:46:46.496368 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:46.496216 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:46:46.496368 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:46.496281 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12f1a157-ce5b-4040-98de-ed53507268db-metrics-certs podName:12f1a157-ce5b-4040-98de-ed53507268db nodeName:}" failed. No retries permitted until 2026-11-04 04:46:50.496261852 +0000 UTC m=+10.081530575 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12f1a157-ce5b-4040-98de-ed53507268db-metrics-certs") pod "network-metrics-daemon-pbjtk" (UID: "12f1a157-ce5b-4040-98de-ed53507268db") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:46:46.598031 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:46.597974 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24bc4691-e47f-4f98-ad47-f14654d95d28-original-pull-secret\") pod \"global-pull-secret-syncer-m5khv\" (UID: \"24bc4691-e47f-4f98-ad47-f14654d95d28\") " pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:46:46.598246 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:46.598196 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 04:46:46.598339 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:46.598301 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24bc4691-e47f-4f98-ad47-f14654d95d28-original-pull-secret podName:24bc4691-e47f-4f98-ad47-f14654d95d28 nodeName:}" failed. No retries permitted until 2026-11-04 04:46:50.59828105 +0000 UTC m=+10.183549767 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24bc4691-e47f-4f98-ad47-f14654d95d28-original-pull-secret") pod "global-pull-secret-syncer-m5khv" (UID: "24bc4691-e47f-4f98-ad47-f14654d95d28") : object "kube-system"/"original-pull-secret" not registered Nov 04 04:46:46.698861 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:46.698832 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct442\" (UniqueName: \"kubernetes.io/projected/d0e2510f-0579-4594-bd3b-a3dadf3fb81a-kube-api-access-ct442\") pod \"network-check-target-6xc6w\" (UID: \"d0e2510f-0579-4594-bd3b-a3dadf3fb81a\") " pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:46:46.699045 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:46.699008 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 04:46:46.699106 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:46.699054 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 04:46:46.699106 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:46.699069 2577 projected.go:196] Error preparing data for projected volume kube-api-access-ct442 for pod openshift-network-diagnostics/network-check-target-6xc6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:46:46.699192 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:46.699153 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0e2510f-0579-4594-bd3b-a3dadf3fb81a-kube-api-access-ct442 podName:d0e2510f-0579-4594-bd3b-a3dadf3fb81a nodeName:}" failed. No retries permitted until 2026-11-04 04:46:50.69913424 +0000 UTC m=+10.284402969 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ct442" (UniqueName: "kubernetes.io/projected/d0e2510f-0579-4594-bd3b-a3dadf3fb81a-kube-api-access-ct442") pod "network-check-target-6xc6w" (UID: "d0e2510f-0579-4594-bd3b-a3dadf3fb81a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:46:46.822155 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:46.822051 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:46:46.822325 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:46.822229 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbjtk" podUID="12f1a157-ce5b-4040-98de-ed53507268db" Nov 04 04:46:46.822325 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:46.822312 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:46:46.822439 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:46.822383 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5khv" podUID="24bc4691-e47f-4f98-ad47-f14654d95d28" Nov 04 04:46:46.822501 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:46.822443 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:46:46.822555 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:46.822509 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xc6w" podUID="d0e2510f-0579-4594-bd3b-a3dadf3fb81a" Nov 04 04:46:46.894622 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:46.894584 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" event={"ID":"81b6936b6f0cd5f6748dab38c3bd34d9","Type":"ContainerStarted","Data":"7ca2e6ec78c5c9ab0e010fe63d0d4a411fd28cbf1cf9b9584e51bf922aff8ff7"} Nov 04 04:46:46.905676 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:46.905645 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal" event={"ID":"e7859ee47f36ca521b3b7f587511c5ea","Type":"ContainerStarted","Data":"2c1cab6bde3566f5877661e42a1dce8cbcbb185d0c7d2edc257c5a214bb2f1c2"} Nov 04 04:46:47.021268 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:47.021205 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" podStartSLOduration=5.021183684 podStartE2EDuration="5.021183684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:47.0210812 +0000 UTC m=+6.606349940" watchObservedRunningTime="2026-11-04 04:46:47.021183684 +0000 UTC m=+6.606452432" Nov 04 04:46:47.914182 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:47.914117 2577 generic.go:356] "Generic (PLEG): container finished" podID="27a76981-142e-4959-aac0-daaffc4f9b95" containerID="68868f5b637266aafa28623fb8a59e4f4494e7d2f457c5e47eb3f3c56cd8ebf1" exitCode=0 Nov 04 04:46:47.915300 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:47.915261 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rkbf" event={"ID":"27a76981-142e-4959-aac0-daaffc4f9b95","Type":"ContainerDied","Data":"68868f5b637266aafa28623fb8a59e4f4494e7d2f457c5e47eb3f3c56cd8ebf1"} Nov 04 04:46:48.150803 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:48.150695 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal" podStartSLOduration=6.150674926 podStartE2EDuration="6.150674926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:47.14252052 +0000 UTC m=+6.727789259" watchObservedRunningTime="2026-11-04 04:46:48.150674926 +0000 UTC m=+7.735943667" Nov 04 04:46:48.822860 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:48.822350 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:46:48.822860 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:48.822537 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbjtk" podUID="12f1a157-ce5b-4040-98de-ed53507268db" Nov 04 04:46:48.822860 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:48.822604 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:46:48.822860 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:48.822678 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5khv" podUID="24bc4691-e47f-4f98-ad47-f14654d95d28" Nov 04 04:46:48.822860 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:48.822717 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:46:48.822860 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:48.822791 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xc6w" podUID="d0e2510f-0579-4594-bd3b-a3dadf3fb81a" Nov 04 04:46:48.925742 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:48.925703 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7csxg" event={"ID":"030a8f10-c4b1-45b2-abb8-88ed48e53a80","Type":"ContainerStarted","Data":"271054ba71d418b328d11bcf3f917ec0abb08b60dc562fac3765f06c4740a1e9"} Nov 04 04:46:49.931002 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:49.930811 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" event={"ID":"26237d52-9119-41f2-919a-5e5153ea223a","Type":"ContainerStarted","Data":"5c789927f9aca59f4ef2babbdd0110763845bb6c253c28cd4ad989b0def3c5ef"} Nov 04 04:46:49.935051 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:49.934900 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lfjtg" event={"ID":"acddcc2f-2b31-478b-a0de-552929cd7bd4","Type":"ContainerStarted","Data":"8fea3d8529cc90698398a248ec30fcfdb2225dc491e94cb2ee554f8c28d270b5"} Nov 04 04:46:50.046217 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:50.046157 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7csxg" podStartSLOduration=4.43672853 podStartE2EDuration="9.046137741s" totalImagesPullingTime="4.609409211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:49.040950514 +0000 UTC m=+8.626219246" watchObservedRunningTime="2026-11-04 04:46:50.046137741 +0000 UTC m=+9.631406476" Nov 04 04:46:50.046407 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:50.046328 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lfjtg" podStartSLOduration=3.074562724 podStartE2EDuration="9.046317115s" totalImagesPullingTime="5.971754391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:50.046069841 +0000 UTC m=+9.631338594" watchObservedRunningTime="2026-11-04 04:46:50.046317115 +0000 UTC m=+9.631585850" Nov 04 04:46:50.100071 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:50.100042 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7csxg" Nov 04 04:46:50.101006 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:50.100801 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7csxg" Nov 04 04:46:50.529736 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:50.529671 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f1a157-ce5b-4040-98de-ed53507268db-metrics-certs\") pod \"network-metrics-daemon-pbjtk\" (UID: \"12f1a157-ce5b-4040-98de-ed53507268db\") " pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:46:50.529912 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:50.529879 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:46:50.530007 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:50.529952 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12f1a157-ce5b-4040-98de-ed53507268db-metrics-certs podName:12f1a157-ce5b-4040-98de-ed53507268db nodeName:}" failed. No retries permitted until 2026-11-04 04:46:58.52993399 +0000 UTC m=+18.115202717 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12f1a157-ce5b-4040-98de-ed53507268db-metrics-certs") pod "network-metrics-daemon-pbjtk" (UID: "12f1a157-ce5b-4040-98de-ed53507268db") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:46:50.630341 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:50.630303 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24bc4691-e47f-4f98-ad47-f14654d95d28-original-pull-secret\") pod \"global-pull-secret-syncer-m5khv\" (UID: \"24bc4691-e47f-4f98-ad47-f14654d95d28\") " pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:46:50.630473 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:50.630450 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 04:46:50.630556 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:50.630511 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24bc4691-e47f-4f98-ad47-f14654d95d28-original-pull-secret podName:24bc4691-e47f-4f98-ad47-f14654d95d28 nodeName:}" failed. No retries permitted until 2026-11-04 04:46:58.630493534 +0000 UTC m=+18.215762355 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24bc4691-e47f-4f98-ad47-f14654d95d28-original-pull-secret") pod "global-pull-secret-syncer-m5khv" (UID: "24bc4691-e47f-4f98-ad47-f14654d95d28") : object "kube-system"/"original-pull-secret" not registered Nov 04 04:46:50.731343 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:50.731284 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct442\" (UniqueName: \"kubernetes.io/projected/d0e2510f-0579-4594-bd3b-a3dadf3fb81a-kube-api-access-ct442\") pod \"network-check-target-6xc6w\" (UID: \"d0e2510f-0579-4594-bd3b-a3dadf3fb81a\") " pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:46:50.731518 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:50.731452 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 04:46:50.731518 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:50.731473 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 04:46:50.731518 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:50.731485 2577 projected.go:196] Error preparing data for projected volume kube-api-access-ct442 for pod openshift-network-diagnostics/network-check-target-6xc6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:46:50.731671 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:50.731532 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0e2510f-0579-4594-bd3b-a3dadf3fb81a-kube-api-access-ct442 podName:d0e2510f-0579-4594-bd3b-a3dadf3fb81a nodeName:}" failed. No retries permitted until 2026-11-04 04:46:58.731518421 +0000 UTC m=+18.316787150 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ct442" (UniqueName: "kubernetes.io/projected/d0e2510f-0579-4594-bd3b-a3dadf3fb81a-kube-api-access-ct442") pod "network-check-target-6xc6w" (UID: "d0e2510f-0579-4594-bd3b-a3dadf3fb81a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:46:50.820107 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:50.820035 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:46:50.820107 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:50.820056 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:46:50.820311 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:50.820166 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5khv" podUID="24bc4691-e47f-4f98-ad47-f14654d95d28" Nov 04 04:46:50.820311 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:50.820194 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:46:50.820408 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:50.820304 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbjtk" podUID="12f1a157-ce5b-4040-98de-ed53507268db" Nov 04 04:46:50.820408 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:50.820354 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xc6w" podUID="d0e2510f-0579-4594-bd3b-a3dadf3fb81a" Nov 04 04:46:51.681960 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:51.681828 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 04:46:51.833946 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:51.833782 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T04:46:51.681846188Z","UUID":"e40af076-f4f8-415e-bd84-8313cdc82f99","Handler":null,"Name":"","Endpoint":""} Nov 04 04:46:51.835467 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:51.835445 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 04:46:51.835595 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:51.835476 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 04:46:51.942469 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:51.942438 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 04:46:51.942973 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:51.942940 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" event={"ID":"26237d52-9119-41f2-919a-5e5153ea223a","Type":"ContainerStarted","Data":"43e33413676e9384bf3a1d8b0f81324c75d05cd7d92e6f3a3cd6eeec547474a1"} Nov 04 04:46:52.819264 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:52.819226 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:46:52.819714 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:52.819368 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbjtk" podUID="12f1a157-ce5b-4040-98de-ed53507268db" Nov 04 04:46:52.819714 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:52.819684 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:46:52.819828 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:52.819749 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5khv" podUID="24bc4691-e47f-4f98-ad47-f14654d95d28" Nov 04 04:46:52.820047 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:52.819900 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:46:52.820047 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:52.820009 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xc6w" podUID="d0e2510f-0579-4594-bd3b-a3dadf3fb81a" Nov 04 04:46:52.947391 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:52.947357 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cxp8f" event={"ID":"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b","Type":"ContainerStarted","Data":"8c9272c6fab6074f9a14f010b14d73aa080cbe7b3712578712344ef64924c926"} Nov 04 04:46:52.949905 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:52.949864 2577 generic.go:356] "Generic (PLEG): container finished" podID="27a76981-142e-4959-aac0-daaffc4f9b95" containerID="cf6e9c76338cd6d6d8b636233f0ff357b51654e656337935d91f8cd39c47df23" exitCode=0 Nov 04 04:46:52.950061 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:52.949964 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rkbf" event={"ID":"27a76981-142e-4959-aac0-daaffc4f9b95","Type":"ContainerDied","Data":"cf6e9c76338cd6d6d8b636233f0ff357b51654e656337935d91f8cd39c47df23"} Nov 04 04:46:52.953184 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:52.953156 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wwxmc" event={"ID":"ed8f18be-cea2-4c8f-a143-25ddd48b1a74","Type":"ContainerStarted","Data":"e131bcfc1868fc1c52d4b01618c1b091929e2618e7e8b77ee8b71b3d05045593"} Nov 04 04:46:53.062419 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:53.062339 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-cxp8f" podStartSLOduration=3.269574107 podStartE2EDuration="12.062320676s" totalImagesPullingTime="8.792746569s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:53.061574985 +0000 UTC m=+12.646843720" watchObservedRunningTime="2026-11-04 04:46:53.062320676 +0000 UTC m=+12.647589414" Nov 04 04:46:53.295704 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:53.295644 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-wwxmc" podStartSLOduration=3.446597377 podStartE2EDuration="12.29563186s" totalImagesPullingTime="8.849034483s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:53.295355108 +0000 UTC m=+12.880623834" watchObservedRunningTime="2026-11-04 04:46:53.29563186 +0000 UTC m=+12.880900594" Nov 04 04:46:53.957563 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:53.957300 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-znffl" event={"ID":"565b853c-b917-41a5-b739-368db09dac1f","Type":"ContainerStarted","Data":"9ae565d6ab32d0fecb3f612e4558d68673f86f97c23b7ef353430b5de35a9172"} Nov 04 04:46:53.961584 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:53.961553 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" event={"ID":"26237d52-9119-41f2-919a-5e5153ea223a","Type":"ContainerStarted","Data":"bf33a425548758594a44ccfe818d83d2f2da66e79ce6a74ae7046a56c626d069"} Nov 04 04:46:54.191138 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:54.191088 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-znffl" podStartSLOduration=4.381157706 podStartE2EDuration="13.19107503s" totalImagesPullingTime="8.809917324s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:54.072485395 +0000 UTC m=+13.657754128" watchObservedRunningTime="2026-11-04 04:46:54.19107503 +0000 UTC m=+13.776343763" Nov 04 04:46:54.191316 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:54.191182 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" podStartSLOduration=3.6605585 podStartE2EDuration="13.191177209s" totalImagesPullingTime="9.530618709s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:46:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:54.190577457 +0000 UTC m=+13.775846188" watchObservedRunningTime="2026-11-04 04:46:54.191177209 +0000 UTC m=+13.776445944" Nov 04 04:46:54.819857 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:54.819769 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:46:54.820025 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:54.819783 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:46:54.820025 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:54.819901 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5khv" podUID="24bc4691-e47f-4f98-ad47-f14654d95d28" Nov 04 04:46:54.820025 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:54.819919 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:46:54.820152 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:54.820063 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbjtk" podUID="12f1a157-ce5b-4040-98de-ed53507268db" Nov 04 04:46:54.820185 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:54.820149 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xc6w" podUID="d0e2510f-0579-4594-bd3b-a3dadf3fb81a" Nov 04 04:46:54.964956 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:54.964918 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hlw45" event={"ID":"63e7d159-5f5f-4802-997b-5bfc47f1dad2","Type":"ContainerStarted","Data":"4b7c7ac03d575f6d9ee1349eda05671f4135bbcbbab63dad5891feb1a14105ea"} Nov 04 04:46:54.967843 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:54.967811 2577 generic.go:356] "Generic (PLEG): container finished" podID="27a76981-142e-4959-aac0-daaffc4f9b95" containerID="cbd740d0a140fda47863875b59d967f86cb183ac33093affc5ead75772815e6c" exitCode=0 Nov 04 04:46:54.967979 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:54.967891 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rkbf" event={"ID":"27a76981-142e-4959-aac0-daaffc4f9b95","Type":"ContainerDied","Data":"cbd740d0a140fda47863875b59d967f86cb183ac33093affc5ead75772815e6c"} Nov 04 04:46:55.080979 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:55.080891 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hlw45" podStartSLOduration=2.757467172 podStartE2EDuration="14.080879077s" totalImagesPullingTime="11.323411905s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:55.080428638 +0000 UTC m=+14.665697372" watchObservedRunningTime="2026-11-04 04:46:55.080879077 +0000 UTC m=+14.666147811" Nov 04 04:46:56.820174 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:56.820141 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:46:56.820911 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:56.820181 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:46:56.820911 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:56.820141 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:46:56.820911 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:56.820279 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbjtk" podUID="12f1a157-ce5b-4040-98de-ed53507268db" Nov 04 04:46:56.820911 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:56.820370 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5khv" podUID="24bc4691-e47f-4f98-ad47-f14654d95d28" Nov 04 04:46:56.820911 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:56.820453 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xc6w" podUID="d0e2510f-0579-4594-bd3b-a3dadf3fb81a" Nov 04 04:46:56.974117 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:56.974080 2577 generic.go:356] "Generic (PLEG): container finished" podID="27a76981-142e-4959-aac0-daaffc4f9b95" containerID="3fb4f84439b6598bb978e92cb5ed2e1ca18e0c86345a8b14fdce3277f4cdb70d" exitCode=0 Nov 04 04:46:56.974287 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:56.974122 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rkbf" event={"ID":"27a76981-142e-4959-aac0-daaffc4f9b95","Type":"ContainerDied","Data":"3fb4f84439b6598bb978e92cb5ed2e1ca18e0c86345a8b14fdce3277f4cdb70d"} Nov 04 04:46:57.371820 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:57.371776 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:46:57.538324 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:57.538289 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:46:57.738816 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:57.738575 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:46:57.854820 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:57.854776 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:46:58.060735 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:58.060660 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7csxg" Nov 04 04:46:58.060907 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:58.060815 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 04:46:58.061532 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:58.061506 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7csxg" Nov 04 04:46:58.228196 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:58.228159 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:46:58.592894 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:58.592852 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f1a157-ce5b-4040-98de-ed53507268db-metrics-certs\") pod \"network-metrics-daemon-pbjtk\" (UID: \"12f1a157-ce5b-4040-98de-ed53507268db\") " pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:46:58.593086 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:58.593027 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:46:58.593140 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:58.593094 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12f1a157-ce5b-4040-98de-ed53507268db-metrics-certs podName:12f1a157-ce5b-4040-98de-ed53507268db nodeName:}" failed. No retries permitted until 2026-11-04 04:47:14.593078569 +0000 UTC m=+34.178347295 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12f1a157-ce5b-4040-98de-ed53507268db-metrics-certs") pod "network-metrics-daemon-pbjtk" (UID: "12f1a157-ce5b-4040-98de-ed53507268db") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:46:58.694104 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:58.694061 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24bc4691-e47f-4f98-ad47-f14654d95d28-original-pull-secret\") pod \"global-pull-secret-syncer-m5khv\" (UID: \"24bc4691-e47f-4f98-ad47-f14654d95d28\") " pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:46:58.694274 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:58.694231 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 04:46:58.694331 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:58.694306 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24bc4691-e47f-4f98-ad47-f14654d95d28-original-pull-secret podName:24bc4691-e47f-4f98-ad47-f14654d95d28 nodeName:}" failed. No retries permitted until 2026-11-04 04:47:14.694284012 +0000 UTC m=+34.279552735 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24bc4691-e47f-4f98-ad47-f14654d95d28-original-pull-secret") pod "global-pull-secret-syncer-m5khv" (UID: "24bc4691-e47f-4f98-ad47-f14654d95d28") : object "kube-system"/"original-pull-secret" not registered Nov 04 04:46:58.794659 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:58.794546 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct442\" (UniqueName: \"kubernetes.io/projected/d0e2510f-0579-4594-bd3b-a3dadf3fb81a-kube-api-access-ct442\") pod \"network-check-target-6xc6w\" (UID: \"d0e2510f-0579-4594-bd3b-a3dadf3fb81a\") " pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:46:58.794776 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:58.794712 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 04:46:58.794776 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:58.794735 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 04:46:58.794776 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:58.794749 2577 projected.go:196] Error preparing data for projected volume kube-api-access-ct442 for pod openshift-network-diagnostics/network-check-target-6xc6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:46:58.794915 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:58.794811 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0e2510f-0579-4594-bd3b-a3dadf3fb81a-kube-api-access-ct442 podName:d0e2510f-0579-4594-bd3b-a3dadf3fb81a nodeName:}" failed. No retries permitted until 2026-11-04 04:47:14.794791208 +0000 UTC m=+34.380059940 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ct442" (UniqueName: "kubernetes.io/projected/d0e2510f-0579-4594-bd3b-a3dadf3fb81a-kube-api-access-ct442") pod "network-check-target-6xc6w" (UID: "d0e2510f-0579-4594-bd3b-a3dadf3fb81a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:46:58.819942 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:58.819796 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:46:58.819942 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:58.819822 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:46:58.820119 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:58.819956 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5khv" podUID="24bc4691-e47f-4f98-ad47-f14654d95d28" Nov 04 04:46:58.820119 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:58.820006 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:46:58.820228 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:58.820115 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbjtk" podUID="12f1a157-ce5b-4040-98de-ed53507268db" Nov 04 04:46:58.820228 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:46:58.820197 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xc6w" podUID="d0e2510f-0579-4594-bd3b-a3dadf3fb81a" Nov 04 04:46:58.982558 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:58.982062 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" event={"ID":"64661f31-04d2-41e7-aab7-28000a68fc64","Type":"ContainerStarted","Data":"4bceff240336e8dbd4c6c1477d9b51712e6c057967be57a927d8b7be2b7b6302"} Nov 04 04:46:58.982558 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:58.982375 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" event={"ID":"64661f31-04d2-41e7-aab7-28000a68fc64","Type":"ContainerStarted","Data":"c1312fcd88a6e2805a8904471a69af62574c0af6c82cdbaec538c348c6129e8f"} Nov 04 04:46:58.982558 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:58.982425 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" event={"ID":"64661f31-04d2-41e7-aab7-28000a68fc64","Type":"ContainerStarted","Data":"52260d7a202d8e54eaa74ba961f323abd7066dd79359eba598d86daa4235a9bb"} Nov 04 04:46:58.983201 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:58.982472 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" event={"ID":"64661f31-04d2-41e7-aab7-28000a68fc64","Type":"ContainerStarted","Data":"4be37cd4382308d3f6bb75c667af448c319ccbc3d7c64f388b9ec91c2ccd311c"} Nov 04 04:46:59.618203 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:59.618164 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:46:59.736077 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:59.736035 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:46:59.986051 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:59.985967 2577 generic.go:356] "Generic (PLEG): container finished" podID="27a76981-142e-4959-aac0-daaffc4f9b95" containerID="372489729f01bb7e98ecc41f4f0a55abf34228f5aa68762a791c3993e22781db" exitCode=0 Nov 04 04:46:59.986051 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:59.986020 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rkbf" event={"ID":"27a76981-142e-4959-aac0-daaffc4f9b95","Type":"ContainerDied","Data":"372489729f01bb7e98ecc41f4f0a55abf34228f5aa68762a791c3993e22781db"} Nov 04 04:46:59.990630 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:59.990608 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" event={"ID":"64661f31-04d2-41e7-aab7-28000a68fc64","Type":"ContainerStarted","Data":"a512e2e0f4d8c2221325a73eca7bb1294bb6efe022e7f30c7b365980124f1f70"} Nov 04 04:46:59.990733 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:46:59.990664 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" event={"ID":"64661f31-04d2-41e7-aab7-28000a68fc64","Type":"ContainerStarted","Data":"ed5b2bc2cb7852c2c2838eaaf2fb38125ad865cf041738647ba7cd574328c243"} Nov 04 04:47:00.820713 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:00.820673 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:47:00.820869 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:00.820764 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xc6w" podUID="d0e2510f-0579-4594-bd3b-a3dadf3fb81a" Nov 04 04:47:00.820951 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:00.820865 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:47:00.820951 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:00.820886 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:47:00.821128 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:00.821105 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5khv" podUID="24bc4691-e47f-4f98-ad47-f14654d95d28" Nov 04 04:47:00.821229 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:00.821210 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbjtk" podUID="12f1a157-ce5b-4040-98de-ed53507268db" Nov 04 04:47:00.995747 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:00.995710 2577 generic.go:356] "Generic (PLEG): container finished" podID="27a76981-142e-4959-aac0-daaffc4f9b95" containerID="7c827f6e5918fc31fd579c6450d512f4e164a8c799c2a32e0e8b34fcfca65aac" exitCode=0 Nov 04 04:47:00.996512 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:00.995760 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rkbf" event={"ID":"27a76981-142e-4959-aac0-daaffc4f9b95","Type":"ContainerDied","Data":"7c827f6e5918fc31fd579c6450d512f4e164a8c799c2a32e0e8b34fcfca65aac"} Nov 04 04:47:01.822242 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:01.822206 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:01.939657 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:01.939627 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:02.000907 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:02.000872 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" event={"ID":"64661f31-04d2-41e7-aab7-28000a68fc64","Type":"ContainerStarted","Data":"a2681d5728bebbe93d6215f7d4ca45559ae0474d37b071cb8c20dca38519751d"} Nov 04 04:47:02.004122 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:02.004099 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rkbf" event={"ID":"27a76981-142e-4959-aac0-daaffc4f9b95","Type":"ContainerStarted","Data":"b4fcceedf8ce94671c7177fb4d50a4c7ff3afb925ed7149936185dbd0e079a84"} Nov 04 04:47:02.121852 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:02.121807 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7rkbf" podStartSLOduration=8.57069922 podStartE2EDuration="21.12179326s" totalImagesPullingTime="12.55109404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:46:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:02.12138936 +0000 UTC m=+21.706658093" watchObservedRunningTime="2026-11-04 04:47:02.12179326 +0000 UTC m=+21.707062006" Nov 04 04:47:02.241858 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:02.241825 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:02.365335 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:02.365305 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:02.560051 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:02.559945 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:02.685347 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:02.685314 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:02.809835 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:02.809799 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:02.819909 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:02.819854 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:47:02.819909 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:02.819873 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:47:02.820027 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:02.819960 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5khv" podUID="24bc4691-e47f-4f98-ad47-f14654d95d28" Nov 04 04:47:02.820027 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:02.820019 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:47:02.820138 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:02.820117 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbjtk" podUID="12f1a157-ce5b-4040-98de-ed53507268db" Nov 04 04:47:02.820224 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:02.820205 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xc6w" podUID="d0e2510f-0579-4594-bd3b-a3dadf3fb81a" Nov 04 04:47:02.927604 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:02.927573 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:03.047815 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:03.047782 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:04.012179 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:04.011945 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" event={"ID":"64661f31-04d2-41e7-aab7-28000a68fc64","Type":"ContainerStarted","Data":"84d4875714afff9f165def1d7d0792c44e585d7a904b553caacb6c3f650b5269"} Nov 04 04:47:04.012365 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:04.012329 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:04.012437 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:04.012407 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:04.020773 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:04.020741 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:04.029294 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:04.029263 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:04.029437 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:04.029390 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:04.139285 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:04.139252 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:04.260614 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:04.260576 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:04.283904 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:04.283804 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" podStartSLOduration=7.780141632 podStartE2EDuration="23.283792325s" totalImagesPullingTime="15.503650693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:04.283315405 +0000 UTC m=+23.868584140" watchObservedRunningTime="2026-11-04 04:47:04.283792325 +0000 UTC m=+23.869061058" Nov 04 04:47:04.819968 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:04.819938 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:47:04.820163 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:04.820001 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:47:04.820163 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:04.819950 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:47:04.820163 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:04.820115 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbjtk" podUID="12f1a157-ce5b-4040-98de-ed53507268db" Nov 04 04:47:04.820329 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:04.820222 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xc6w" podUID="d0e2510f-0579-4594-bd3b-a3dadf3fb81a" Nov 04 04:47:04.820329 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:04.820271 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5khv" podUID="24bc4691-e47f-4f98-ad47-f14654d95d28" Nov 04 04:47:05.013921 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:05.013894 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 04:47:05.320246 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:05.320204 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:05.439924 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:05.439891 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:05.691206 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:05.691177 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:05.924703 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:05.924669 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pbjtk"] Nov 04 04:47:05.924860 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:05.924808 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:47:05.924921 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:05.924907 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbjtk" podUID="12f1a157-ce5b-4040-98de-ed53507268db" Nov 04 04:47:05.927539 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:05.927513 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6xc6w"] Nov 04 04:47:05.927637 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:05.927620 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:47:05.927708 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:05.927694 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xc6w" podUID="d0e2510f-0579-4594-bd3b-a3dadf3fb81a" Nov 04 04:47:05.974563 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:05.974537 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m5khv"] Nov 04 04:47:05.974716 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:05.974639 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:47:05.974755 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:05.974719 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5khv" podUID="24bc4691-e47f-4f98-ad47-f14654d95d28" Nov 04 04:47:06.588281 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:06.588253 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wwxmc_ed8f18be-cea2-4c8f-a143-25ddd48b1a74/dns-node-resolver/0.log" Nov 04 04:47:06.817640 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:06.817606 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:07.038549 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:07.038517 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:07.159045 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:07.159014 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:07.414922 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:07.414873 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:07.650099 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:07.650072 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lfjtg_acddcc2f-2b31-478b-a0de-552929cd7bd4/node-ca/0.log" Nov 04 04:47:07.819256 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:07.819222 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:47:07.819436 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:07.819339 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:47:07.819436 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:07.819355 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xc6w" podUID="d0e2510f-0579-4594-bd3b-a3dadf3fb81a" Nov 04 04:47:07.819436 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:07.819372 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:47:07.819586 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:07.819458 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbjtk" podUID="12f1a157-ce5b-4040-98de-ed53507268db" Nov 04 04:47:07.819586 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:07.819521 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5khv" podUID="24bc4691-e47f-4f98-ad47-f14654d95d28" Nov 04 04:47:07.932479 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:07.932451 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:08.082721 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:08.082685 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:08.199695 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:08.199622 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:08.401978 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:08.401943 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:08.609996 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:08.609967 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:09.202966 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:09.202934 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:09.403078 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:09.403048 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:09.807219 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:09.807180 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:09.819147 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:09.819115 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:47:09.819290 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:09.819157 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:47:09.819290 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:09.819157 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:47:09.819290 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:09.819223 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xc6w" podUID="d0e2510f-0579-4594-bd3b-a3dadf3fb81a" Nov 04 04:47:09.819394 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:09.819366 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5khv" podUID="24bc4691-e47f-4f98-ad47-f14654d95d28" Nov 04 04:47:09.819467 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:09.819449 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbjtk" podUID="12f1a157-ce5b-4040-98de-ed53507268db" Nov 04 04:47:10.002862 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:10.002818 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:10.601694 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:10.601646 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:11.203001 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:11.202954 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:11.433909 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:11.433868 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:11.548512 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:11.548430 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:11.805830 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:11.805753 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:11.819768 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:11.819734 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:47:11.819926 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:11.819837 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:47:11.819926 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:11.819850 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xc6w" podUID="d0e2510f-0579-4594-bd3b-a3dadf3fb81a" Nov 04 04:47:11.819926 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:11.819870 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:47:11.820104 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:11.819964 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5khv" podUID="24bc4691-e47f-4f98-ad47-f14654d95d28" Nov 04 04:47:11.820104 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:11.820073 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbjtk" podUID="12f1a157-ce5b-4040-98de-ed53507268db" Nov 04 04:47:11.979678 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:11.979643 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:12.099391 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.099307 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:12.245308 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.245280 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeReady" Nov 04 04:47:12.245464 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.245403 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 04 04:47:12.357949 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.357928 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-q7xkm"] Nov 04 04:47:12.362239 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.362215 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-t9tvv"] Nov 04 04:47:12.362368 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.362330 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q7xkm" Nov 04 04:47:12.364917 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.364896 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-724sb"] Nov 04 04:47:12.365044 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.365032 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t9tvv" Nov 04 04:47:12.367932 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.367914 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:12.415346 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.415313 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6sxpx\"" Nov 04 04:47:12.415481 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.415362 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 04:47:12.415481 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.415401 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 04:47:12.416903 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.416886 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 04:47:12.417177 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.417161 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-q5r74\"" Nov 04 04:47:12.417177 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.417167 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 04:47:12.417285 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.417229 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 04:47:12.422535 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.422521 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 04:47:12.431790 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.431773 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t9tvv"] Nov 04 04:47:12.435050 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.435026 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q7xkm"] Nov 04 04:47:12.471243 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.471220 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 04:47:12.471410 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.471304 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 04:47:12.471410 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.471369 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 04:47:12.471523 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.471507 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-724sb"] Nov 04 04:47:12.496651 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.496626 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3e3cff9-0366-457e-a337-32722ca38e00-tmp-dir\") pod \"dns-default-q7xkm\" (UID: \"c3e3cff9-0366-457e-a337-32722ca38e00\") " pod="openshift-dns/dns-default-q7xkm" Nov 04 04:47:12.496743 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.496654 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjpz9\" (UniqueName: \"kubernetes.io/projected/3bb73603-5b6f-4a77-a39f-033b1f80059f-kube-api-access-sjpz9\") pod \"ingress-canary-t9tvv\" (UID: \"3bb73603-5b6f-4a77-a39f-033b1f80059f\") " pod="openshift-ingress-canary/ingress-canary-t9tvv" Nov 04 04:47:12.496743 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.496681 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e407d116-de8e-4a31-bcee-88546ae61f57-crio-socket\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:12.496830 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.496757 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e407d116-de8e-4a31-bcee-88546ae61f57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:12.496830 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.496790 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e3cff9-0366-457e-a337-32722ca38e00-config-volume\") pod \"dns-default-q7xkm\" (UID: \"c3e3cff9-0366-457e-a337-32722ca38e00\") " pod="openshift-dns/dns-default-q7xkm" Nov 04 04:47:12.496830 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.496808 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5s2h\" (UniqueName: \"kubernetes.io/projected/c3e3cff9-0366-457e-a337-32722ca38e00-kube-api-access-p5s2h\") pod \"dns-default-q7xkm\" (UID: \"c3e3cff9-0366-457e-a337-32722ca38e00\") " pod="openshift-dns/dns-default-q7xkm" Nov 04 04:47:12.496830 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.496823 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3bb73603-5b6f-4a77-a39f-033b1f80059f-cert\") pod \"ingress-canary-t9tvv\" (UID: \"3bb73603-5b6f-4a77-a39f-033b1f80059f\") " pod="openshift-ingress-canary/ingress-canary-t9tvv" Nov 04 04:47:12.496949 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.496894 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e407d116-de8e-4a31-bcee-88546ae61f57-data-volume\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:12.496949 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.496915 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9rtr\" (UniqueName: \"kubernetes.io/projected/e407d116-de8e-4a31-bcee-88546ae61f57-kube-api-access-s9rtr\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:12.497036 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.496947 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3e3cff9-0366-457e-a337-32722ca38e00-metrics-tls\") pod \"dns-default-q7xkm\" (UID: \"c3e3cff9-0366-457e-a337-32722ca38e00\") " pod="openshift-dns/dns-default-q7xkm" Nov 04 04:47:12.497036 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.497018 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e407d116-de8e-4a31-bcee-88546ae61f57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:12.515603 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.515583 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jvrn2\"" Nov 04 04:47:12.598114 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.598086 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3e3cff9-0366-457e-a337-32722ca38e00-tmp-dir\") pod \"dns-default-q7xkm\" (UID: \"c3e3cff9-0366-457e-a337-32722ca38e00\") " pod="openshift-dns/dns-default-q7xkm" Nov 04 04:47:12.598114 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.598113 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjpz9\" (UniqueName: \"kubernetes.io/projected/3bb73603-5b6f-4a77-a39f-033b1f80059f-kube-api-access-sjpz9\") pod \"ingress-canary-t9tvv\" (UID: \"3bb73603-5b6f-4a77-a39f-033b1f80059f\") " pod="openshift-ingress-canary/ingress-canary-t9tvv" Nov 04 04:47:12.598298 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.598134 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e407d116-de8e-4a31-bcee-88546ae61f57-crio-socket\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:12.598298 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.598150 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e407d116-de8e-4a31-bcee-88546ae61f57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:12.598298 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.598197 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e3cff9-0366-457e-a337-32722ca38e00-config-volume\") pod \"dns-default-q7xkm\" (UID: \"c3e3cff9-0366-457e-a337-32722ca38e00\") " pod="openshift-dns/dns-default-q7xkm" Nov 04 04:47:12.598298 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.598235 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5s2h\" (UniqueName: \"kubernetes.io/projected/c3e3cff9-0366-457e-a337-32722ca38e00-kube-api-access-p5s2h\") pod \"dns-default-q7xkm\" (UID: \"c3e3cff9-0366-457e-a337-32722ca38e00\") " pod="openshift-dns/dns-default-q7xkm" Nov 04 04:47:12.598298 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.598259 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3bb73603-5b6f-4a77-a39f-033b1f80059f-cert\") pod \"ingress-canary-t9tvv\" (UID: \"3bb73603-5b6f-4a77-a39f-033b1f80059f\") " pod="openshift-ingress-canary/ingress-canary-t9tvv" Nov 04 04:47:12.598523 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.598304 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e407d116-de8e-4a31-bcee-88546ae61f57-data-volume\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:12.598523 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.598332 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e407d116-de8e-4a31-bcee-88546ae61f57-crio-socket\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:12.598523 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.598498 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3e3cff9-0366-457e-a337-32722ca38e00-tmp-dir\") pod \"dns-default-q7xkm\" (UID: \"c3e3cff9-0366-457e-a337-32722ca38e00\") " pod="openshift-dns/dns-default-q7xkm" Nov 04 04:47:12.598664 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.598602 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e407d116-de8e-4a31-bcee-88546ae61f57-data-volume\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:12.598664 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.598328 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9rtr\" (UniqueName: \"kubernetes.io/projected/e407d116-de8e-4a31-bcee-88546ae61f57-kube-api-access-s9rtr\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:12.598664 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.598655 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3e3cff9-0366-457e-a337-32722ca38e00-metrics-tls\") pod \"dns-default-q7xkm\" (UID: \"c3e3cff9-0366-457e-a337-32722ca38e00\") " pod="openshift-dns/dns-default-q7xkm" Nov 04 04:47:12.598791 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.598703 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e407d116-de8e-4a31-bcee-88546ae61f57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:12.598840 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:12.598818 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 04:47:12.598890 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.598861 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e407d116-de8e-4a31-bcee-88546ae61f57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:12.598890 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.598866 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e3cff9-0366-457e-a337-32722ca38e00-config-volume\") pod \"dns-default-q7xkm\" (UID: \"c3e3cff9-0366-457e-a337-32722ca38e00\") " pod="openshift-dns/dns-default-q7xkm" Nov 04 04:47:12.598890 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:12.598882 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e407d116-de8e-4a31-bcee-88546ae61f57-insights-runtime-extractor-tls podName:e407d116-de8e-4a31-bcee-88546ae61f57 nodeName:}" failed. No retries permitted until 2026-11-04 04:47:13.09886288 +0000 UTC m=+32.684131593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e407d116-de8e-4a31-bcee-88546ae61f57-insights-runtime-extractor-tls") pod "insights-runtime-extractor-724sb" (UID: "e407d116-de8e-4a31-bcee-88546ae61f57") : secret "insights-runtime-extractor-tls" not found Nov 04 04:47:12.602236 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.602215 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3e3cff9-0366-457e-a337-32722ca38e00-metrics-tls\") pod \"dns-default-q7xkm\" (UID: \"c3e3cff9-0366-457e-a337-32722ca38e00\") " pod="openshift-dns/dns-default-q7xkm" Nov 04 04:47:12.602330 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.602317 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3bb73603-5b6f-4a77-a39f-033b1f80059f-cert\") pod \"ingress-canary-t9tvv\" (UID: \"3bb73603-5b6f-4a77-a39f-033b1f80059f\") " pod="openshift-ingress-canary/ingress-canary-t9tvv" Nov 04 04:47:12.612273 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.612216 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:12.656061 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.656032 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjpz9\" (UniqueName: \"kubernetes.io/projected/3bb73603-5b6f-4a77-a39f-033b1f80059f-kube-api-access-sjpz9\") pod \"ingress-canary-t9tvv\" (UID: \"3bb73603-5b6f-4a77-a39f-033b1f80059f\") " pod="openshift-ingress-canary/ingress-canary-t9tvv" Nov 04 04:47:12.659400 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.659372 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9rtr\" (UniqueName: \"kubernetes.io/projected/e407d116-de8e-4a31-bcee-88546ae61f57-kube-api-access-s9rtr\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:12.659513 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.659494 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5s2h\" (UniqueName: \"kubernetes.io/projected/c3e3cff9-0366-457e-a337-32722ca38e00-kube-api-access-p5s2h\") pod \"dns-default-q7xkm\" (UID: \"c3e3cff9-0366-457e-a337-32722ca38e00\") " pod="openshift-dns/dns-default-q7xkm" Nov 04 04:47:12.678418 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.678399 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q7xkm" Nov 04 04:47:12.681483 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.681466 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t9tvv" Nov 04 04:47:12.782285 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.782259 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:12.912114 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:12.912032 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:13.069867 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:13.069838 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:13.102177 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:13.102154 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e407d116-de8e-4a31-bcee-88546ae61f57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:13.105051 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:13.102541 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 04:47:13.105051 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:13.102640 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e407d116-de8e-4a31-bcee-88546ae61f57-insights-runtime-extractor-tls podName:e407d116-de8e-4a31-bcee-88546ae61f57 nodeName:}" failed. No retries permitted until 2026-11-04 04:47:14.102608949 +0000 UTC m=+33.687877679 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e407d116-de8e-4a31-bcee-88546ae61f57-insights-runtime-extractor-tls") pod "insights-runtime-extractor-724sb" (UID: "e407d116-de8e-4a31-bcee-88546ae61f57") : secret "insights-runtime-extractor-tls" not found Nov 04 04:47:13.113385 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:13.113333 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t9tvv"] Nov 04 04:47:13.116528 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:13.116507 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q7xkm"] Nov 04 04:47:13.117392 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:47:13.117301 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb73603_5b6f_4a77_a39f_033b1f80059f.slice/crio-bdf831998fa83bbf9a15393773f1db7c81190256b00a0f1ceeff03dc4451b05c WatchSource:0}: Error finding container bdf831998fa83bbf9a15393773f1db7c81190256b00a0f1ceeff03dc4451b05c: Status 404 returned error can't find the container with id bdf831998fa83bbf9a15393773f1db7c81190256b00a0f1ceeff03dc4451b05c Nov 04 04:47:13.119163 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:47:13.119143 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e3cff9_0366_457e_a337_32722ca38e00.slice/crio-bb0bf32ff56b9b965f8693e29ac897a5a843a5597237f7458aa5a5db509f71d3 WatchSource:0}: Error finding container bb0bf32ff56b9b965f8693e29ac897a5a843a5597237f7458aa5a5db509f71d3: Status 404 returned error can't find the container with id bb0bf32ff56b9b965f8693e29ac897a5a843a5597237f7458aa5a5db509f71d3 Nov 04 04:47:13.217952 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:13.217882 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:13.406783 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:13.406755 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:13.529262 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:13.529185 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:13.808785 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:13.808715 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:13.819714 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:13.819695 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:47:13.819760 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:13.819724 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:47:13.819833 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:13.819820 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:47:13.871919 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:13.871894 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 04:47:13.871919 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:13.871894 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 04:47:13.872752 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:13.872734 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 04:47:13.922000 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:13.921966 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-n7w7t\"" Nov 04 04:47:13.922286 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:13.922004 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-w4xfx\"" Nov 04 04:47:13.922286 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:13.922031 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 04:47:13.922286 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:13.922077 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mrmbg\"" Nov 04 04:47:14.007467 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:14.007425 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:14.035849 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:14.035812 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t9tvv" event={"ID":"3bb73603-5b6f-4a77-a39f-033b1f80059f","Type":"ContainerStarted","Data":"bdf831998fa83bbf9a15393773f1db7c81190256b00a0f1ceeff03dc4451b05c"} Nov 04 04:47:14.037331 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:14.037303 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q7xkm" event={"ID":"c3e3cff9-0366-457e-a337-32722ca38e00","Type":"ContainerStarted","Data":"bb0bf32ff56b9b965f8693e29ac897a5a843a5597237f7458aa5a5db509f71d3"} Nov 04 04:47:14.109934 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:14.109891 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e407d116-de8e-4a31-bcee-88546ae61f57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:14.110114 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:14.110046 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 04:47:14.110193 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:14.110123 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e407d116-de8e-4a31-bcee-88546ae61f57-insights-runtime-extractor-tls podName:e407d116-de8e-4a31-bcee-88546ae61f57 nodeName:}" failed. No retries permitted until 2026-11-04 04:47:16.110102809 +0000 UTC m=+35.695371535 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e407d116-de8e-4a31-bcee-88546ae61f57-insights-runtime-extractor-tls") pod "insights-runtime-extractor-724sb" (UID: "e407d116-de8e-4a31-bcee-88546ae61f57") : secret "insights-runtime-extractor-tls" not found Nov 04 04:47:14.408069 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:14.407910 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:14.614411 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:14.614374 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f1a157-ce5b-4040-98de-ed53507268db-metrics-certs\") pod \"network-metrics-daemon-pbjtk\" (UID: \"12f1a157-ce5b-4040-98de-ed53507268db\") " pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:47:14.617381 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:14.617350 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f1a157-ce5b-4040-98de-ed53507268db-metrics-certs\") pod \"network-metrics-daemon-pbjtk\" (UID: \"12f1a157-ce5b-4040-98de-ed53507268db\") " pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:47:14.716051 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:14.715490 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24bc4691-e47f-4f98-ad47-f14654d95d28-original-pull-secret\") pod \"global-pull-secret-syncer-m5khv\" (UID: \"24bc4691-e47f-4f98-ad47-f14654d95d28\") " pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:47:14.725071 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:14.725042 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24bc4691-e47f-4f98-ad47-f14654d95d28-original-pull-secret\") pod \"global-pull-secret-syncer-m5khv\" (UID: \"24bc4691-e47f-4f98-ad47-f14654d95d28\") " pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:47:14.738002 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:14.737960 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:47:14.745250 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:14.745227 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:47:14.818806 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:14.818778 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct442\" (UniqueName: \"kubernetes.io/projected/d0e2510f-0579-4594-bd3b-a3dadf3fb81a-kube-api-access-ct442\") pod \"network-check-target-6xc6w\" (UID: \"d0e2510f-0579-4594-bd3b-a3dadf3fb81a\") " pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:47:14.826666 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:14.826547 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct442\" (UniqueName: \"kubernetes.io/projected/d0e2510f-0579-4594-bd3b-a3dadf3fb81a-kube-api-access-ct442\") pod \"network-check-target-6xc6w\" (UID: \"d0e2510f-0579-4594-bd3b-a3dadf3fb81a\") " pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:47:15.034420 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:15.034349 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:47:15.127747 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:15.127719 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m5khv"] Nov 04 04:47:15.131510 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:47:15.131486 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24bc4691_e47f_4f98_ad47_f14654d95d28.slice/crio-aebd3899e7827a79b9a2d9833edc445155cf66b5d9c1f5f58aebf985c564bf6c WatchSource:0}: Error finding container aebd3899e7827a79b9a2d9833edc445155cf66b5d9c1f5f58aebf985c564bf6c: Status 404 returned error can't find the container with id aebd3899e7827a79b9a2d9833edc445155cf66b5d9c1f5f58aebf985c564bf6c Nov 04 04:47:15.181467 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:15.181413 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pbjtk"] Nov 04 04:47:15.185148 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:15.185124 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:15.187021 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:47:15.186810 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12f1a157_ce5b_4040_98de_ed53507268db.slice/crio-1877e32e770417de592c93724586fd08b55c01287cdef0826be75b746840cc31 WatchSource:0}: Error finding container 1877e32e770417de592c93724586fd08b55c01287cdef0826be75b746840cc31: Status 404 returned error can't find the container with id 1877e32e770417de592c93724586fd08b55c01287cdef0826be75b746840cc31 Nov 04 04:47:15.299120 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:15.299090 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:15.422062 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:15.422031 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6xc6w"] Nov 04 04:47:15.425680 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:47:15.425647 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e2510f_0579_4594_bd3b_a3dadf3fb81a.slice/crio-6fcdfae34ced4daf239c88b5da340350a314a358d75057c991e797ee9d5af075 WatchSource:0}: Error finding container 6fcdfae34ced4daf239c88b5da340350a314a358d75057c991e797ee9d5af075: Status 404 returned error can't find the container with id 6fcdfae34ced4daf239c88b5da340350a314a358d75057c991e797ee9d5af075 Nov 04 04:47:15.606947 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:15.606918 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:15.811402 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:15.811318 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:15.937212 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:15.937172 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:16.044733 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:16.044699 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q7xkm" event={"ID":"c3e3cff9-0366-457e-a337-32722ca38e00","Type":"ContainerStarted","Data":"38dc5ce5f04f6e8b0484db893d17ba53332f24c0dd9d8b1b19dd75ce2bc593d7"} Nov 04 04:47:16.045149 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:16.044752 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q7xkm" event={"ID":"c3e3cff9-0366-457e-a337-32722ca38e00","Type":"ContainerStarted","Data":"c384910626d581d4648cd741fe10ce76bfb69b7d7a0c64a3d85d7eb51749c881"} Nov 04 04:47:16.045149 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:16.045093 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-q7xkm" Nov 04 04:47:16.046703 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:16.046677 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6xc6w" event={"ID":"d0e2510f-0579-4594-bd3b-a3dadf3fb81a","Type":"ContainerStarted","Data":"6fcdfae34ced4daf239c88b5da340350a314a358d75057c991e797ee9d5af075"} Nov 04 04:47:16.048099 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:16.048070 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m5khv" event={"ID":"24bc4691-e47f-4f98-ad47-f14654d95d28","Type":"ContainerStarted","Data":"aebd3899e7827a79b9a2d9833edc445155cf66b5d9c1f5f58aebf985c564bf6c"} Nov 04 04:47:16.049348 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:16.049320 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pbjtk" event={"ID":"12f1a157-ce5b-4040-98de-ed53507268db","Type":"ContainerStarted","Data":"1877e32e770417de592c93724586fd08b55c01287cdef0826be75b746840cc31"} Nov 04 04:47:16.070330 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:16.070254 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:16.129809 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:16.129781 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e407d116-de8e-4a31-bcee-88546ae61f57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:16.130024 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:16.130006 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 04:47:16.130105 ip-10-0-141-132 kubenswrapper[2577]: E1104 04:47:16.130094 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e407d116-de8e-4a31-bcee-88546ae61f57-insights-runtime-extractor-tls podName:e407d116-de8e-4a31-bcee-88546ae61f57 nodeName:}" failed. No retries permitted until 2026-11-04 04:47:20.130060108 +0000 UTC m=+39.715328841 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e407d116-de8e-4a31-bcee-88546ae61f57-insights-runtime-extractor-tls") pod "insights-runtime-extractor-724sb" (UID: "e407d116-de8e-4a31-bcee-88546ae61f57") : secret "insights-runtime-extractor-tls" not found Nov 04 04:47:16.445575 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:16.445542 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:17.056111 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:17.056063 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t9tvv" event={"ID":"3bb73603-5b6f-4a77-a39f-033b1f80059f","Type":"ContainerStarted","Data":"5fe87842f627b1185405468837ec1ba25e6955f50205c17851001d60bcdb247a"} Nov 04 04:47:17.170235 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:17.170177 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-t9tvv" podStartSLOduration=1.800264329 podStartE2EDuration="5.170158474s" totalImagesPullingTime="3.369894145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:17.170110219 +0000 UTC m=+36.755378981" watchObservedRunningTime="2026-11-04 04:47:17.170158474 +0000 UTC m=+36.755427209" Nov 04 04:47:17.170451 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:17.170430 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-q7xkm" podStartSLOduration=2.776035406 podStartE2EDuration="5.170422207s" totalImagesPullingTime="2.394386801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:16.164829669 +0000 UTC m=+35.750098394" watchObservedRunningTime="2026-11-04 04:47:17.170422207 +0000 UTC m=+36.755690941" Nov 04 04:47:17.207219 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:17.207131 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:17.608524 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:17.608495 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:17.804393 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:17.804360 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:18.007667 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:18.007591 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:18.061938 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:18.061843 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pbjtk" event={"ID":"12f1a157-ce5b-4040-98de-ed53507268db","Type":"ContainerStarted","Data":"88394731a68313c4a16f190770b87b7c356f08c0d49a403b5f4252a66c0a980c"} Nov 04 04:47:18.062357 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:18.061970 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pbjtk" event={"ID":"12f1a157-ce5b-4040-98de-ed53507268db","Type":"ContainerStarted","Data":"408383ff7b5e2ef3876b63491583483746958ba8b539234cb03b27a6c0b2e63f"} Nov 04 04:47:18.177207 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:18.177161 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pbjtk" podStartSLOduration=35.080659884 podStartE2EDuration="37.177145318s" totalImagesPullingTime="2.096485434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:18.175972202 +0000 UTC m=+37.761240932" watchObservedRunningTime="2026-11-04 04:47:18.177145318 +0000 UTC m=+37.762414052" Nov 04 04:47:18.210298 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:18.210264 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:19.328814 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:19.328771 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:19.553100 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:19.553070 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:19.670543 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:19.670509 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:19.823356 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:19.823299 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:19.944656 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:19.944565 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:20.069150 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:20.069120 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6xc6w" event={"ID":"d0e2510f-0579-4594-bd3b-a3dadf3fb81a","Type":"ContainerStarted","Data":"1ff73dad7c55b57165fb98f68b49c13c3f97a7f13ea4e9e0cb775d26a5dc0a29"} Nov 04 04:47:20.069339 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:20.069273 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:47:20.159441 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:20.159406 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e407d116-de8e-4a31-bcee-88546ae61f57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:20.166420 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:20.166389 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e407d116-de8e-4a31-bcee-88546ae61f57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:20.183259 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:20.183215 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6xc6w" podStartSLOduration=35.082390436 podStartE2EDuration="39.183200576s" totalImagesPullingTime="4.10081014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:20.182572137 +0000 UTC m=+39.767840871" watchObservedRunningTime="2026-11-04 04:47:20.183200576 +0000 UTC m=+39.768469309" Nov 04 04:47:20.190853 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:20.190820 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:20.584898 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:20.584826 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-724sb"] Nov 04 04:47:20.593078 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:20.593052 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:20.691757 ip-10-0-141-132 kubenswrapper[2577]: W1104 04:47:20.691717 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode407d116_de8e_4a31_bcee_88546ae61f57.slice/crio-eb064591ea1b60ade7aeb414a546736585ead71970e8ca8005e7e75d9b225e72 WatchSource:0}: Error finding container eb064591ea1b60ade7aeb414a546736585ead71970e8ca8005e7e75d9b225e72: Status 404 returned error can't find the container with id eb064591ea1b60ade7aeb414a546736585ead71970e8ca8005e7e75d9b225e72 Nov 04 04:47:20.773548 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:20.773517 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:20.904067 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:20.903973 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:21.014050 ip-10-0-141-132 kubenswrapper[2577]: I1104 04:47:21.014021 2577 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 04:47:21.014199 ip-10-0-141-132 systemd[1]: Stopping Kubernetes Kubelet... Nov 04 04:47:21.036413 ip-10-0-141-132 systemd[1]: kubelet.service: Deactivated successfully. Nov 04 04:47:21.036584 ip-10-0-141-132 systemd[1]: Stopped Kubernetes Kubelet. Nov 04 04:47:21.036632 ip-10-0-141-132 systemd[1]: kubelet.service: Consumed 2.528s CPU time. Nov 04 04:47:21.046662 ip-10-0-141-132 systemd[1]: Starting Kubernetes Kubelet... Nov 04 04:47:21.139129 ip-10-0-141-132 kubenswrapper[4581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 04:47:21.139129 ip-10-0-141-132 kubenswrapper[4581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 04:47:21.139129 ip-10-0-141-132 kubenswrapper[4581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 04:47:21.139129 ip-10-0-141-132 kubenswrapper[4581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 04:47:21.141303 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141215 4581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:47:21.141303 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141303 4581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:47:21.141303 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141308 4581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:47:21.141434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141311 4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:47:21.141434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141314 4581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:47:21.141434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141317 4581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:47:21.141434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141320 4581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:47:21.141434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141328 4581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:47:21.141434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141333 4581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:47:21.141434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141337 4581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:47:21.141434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141341 4581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:47:21.141434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141345 4581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:47:21.141434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141348 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:47:21.141434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141351 4581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:47:21.141434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141353 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:47:21.141434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141356 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:47:21.141434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141359 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:47:21.141434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141361 4581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:47:21.141434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141364 4581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:47:21.141434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141366 4581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:47:21.141434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141369 4581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:47:21.141434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141371 4581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:47:21.141933 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141374 4581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:47:21.141933 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141377 4581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:47:21.141933 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141379 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:47:21.141933 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141382 4581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:47:21.141933 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141384 4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:47:21.141933 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141387 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:47:21.141933 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141390 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:47:21.141933 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141392 4581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:47:21.141933 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141395 4581 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:47:21.141933 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141397 4581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:47:21.141933 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141399 4581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:47:21.141933 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141402 4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:47:21.141933 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141405 4581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:47:21.141933 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141408 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:47:21.141933 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141410 4581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:47:21.141933 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141413 4581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:47:21.141933 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141415 4581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:47:21.141933 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141424 4581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:47:21.141933 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141427 4581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:47:21.141933 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141430 4581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:47:21.142451 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141432 4581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:47:21.142451 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141435 4581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:47:21.142451 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141438 4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:47:21.142451 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141440 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:47:21.142451 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141443 4581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:47:21.142451 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141446 4581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:47:21.142451 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141448 4581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:47:21.142451 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141451 4581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:47:21.142451 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141453 4581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:47:21.142451 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141456 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:47:21.142451 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141458 4581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:47:21.142451 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141461 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:47:21.142451 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141465 4581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:47:21.142451 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141467 4581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:47:21.142451 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141470 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:47:21.142451 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141473 4581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:47:21.142451 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141476 4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:47:21.142451 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141479 4581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:47:21.142451 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141482 4581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:47:21.142908 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141486 4581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:47:21.142908 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141490 4581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:47:21.142908 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141494 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:47:21.142908 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141497 4581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:47:21.142908 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141499 4581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:47:21.142908 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141502 4581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:47:21.142908 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141504 4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:47:21.142908 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141507 4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:47:21.142908 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141509 4581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:47:21.142908 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141512 4581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:47:21.142908 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141514 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:47:21.142908 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141517 4581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:47:21.142908 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141520 4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:47:21.142908 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141523 4581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:47:21.142908 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141526 4581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:47:21.142908 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141528 4581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:47:21.142908 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141531 4581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:47:21.142908 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141533 4581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:47:21.142908 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141536 4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:47:21.143420 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141539 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:47:21.143420 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141542 4581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:47:21.143420 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141544 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:47:21.143420 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141547 4581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:47:21.143420 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141549 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:47:21.143420 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141551 4581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:47:21.143420 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141554 4581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:47:21.143420 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141557 4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:47:21.143420 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141559 4581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:47:21.143420 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141561 4581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:47:21.143420 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141564 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:47:21.143420 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141567 4581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:47:21.143420 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141570 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:47:21.143420 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141573 4581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:47:21.143420 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141576 4581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:47:21.143420 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141578 4581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:47:21.143420 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141581 4581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:47:21.143420 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141583 4581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:47:21.143420 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141586 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:47:21.143420 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141588 4581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:47:21.143934 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141590 4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:47:21.143934 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141598 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:47:21.143934 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141600 4581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:47:21.143934 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141603 4581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:47:21.143934 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141606 4581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:47:21.143934 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.141609 4581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:47:21.143934 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142475 4581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:47:21.143934 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142484 4581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:47:21.143934 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142487 4581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:47:21.143934 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142490 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:47:21.143934 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142493 4581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:47:21.143934 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142496 4581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:47:21.143934 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142499 4581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:47:21.143934 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142501 4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:47:21.143934 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142504 4581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:47:21.143934 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142506 4581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:47:21.143934 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142509 4581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:47:21.143934 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142511 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:47:21.143934 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142514 4581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:47:21.143934 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142517 4581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:47:21.144439 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142519 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:47:21.144439 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142522 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:47:21.144439 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142525 4581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:47:21.144439 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142528 4581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:47:21.144439 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142531 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:47:21.144439 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142534 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:47:21.144439 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142537 4581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:47:21.144439 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142539 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:47:21.144439 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142542 4581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:47:21.144439 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142544 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:47:21.144439 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142547 4581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:47:21.144439 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142549 4581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:47:21.144439 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142552 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:47:21.144439 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142556 4581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:47:21.144439 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142561 4581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:47:21.144439 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142564 4581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:47:21.144439 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142567 4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:47:21.144439 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142570 4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:47:21.144439 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142574 4581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:47:21.144898 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142576 4581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:47:21.144898 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142579 4581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:47:21.144898 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142582 4581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:47:21.144898 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142584 4581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:47:21.144898 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142587 4581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:47:21.144898 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142589 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:47:21.144898 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142592 4581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:47:21.144898 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142594 4581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:47:21.144898 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142597 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:47:21.144898 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142599 4581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:47:21.144898 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142602 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:47:21.144898 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142605 4581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:47:21.144898 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142608 4581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:47:21.144898 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142611 4581 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:47:21.144898 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142614 4581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:47:21.144898 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142617 4581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:47:21.144898 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142621 4581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:47:21.144898 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142624 4581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:47:21.144898 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142626 4581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:47:21.145388 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142629 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:47:21.145388 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142631 4581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:47:21.145388 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142634 4581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:47:21.145388 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142638 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:47:21.145388 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142640 4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:47:21.145388 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142643 4581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:47:21.145388 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142645 4581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:47:21.145388 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142650 4581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:47:21.145388 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142652 4581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:47:21.145388 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142655 4581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:47:21.145388 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142657 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:47:21.145388 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142660 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:47:21.145388 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142663 4581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:47:21.145388 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142666 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:47:21.145388 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142668 4581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:47:21.145388 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142671 4581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:47:21.145388 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142673 4581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:47:21.145388 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142676 4581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:47:21.145388 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142678 4581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:47:21.145849 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142681 4581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:47:21.145849 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142683 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:47:21.145849 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142685 4581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:47:21.145849 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142688 4581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:47:21.145849 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142691 4581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:47:21.145849 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142694 4581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:47:21.145849 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142700 4581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:47:21.145849 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142702 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:47:21.145849 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142705 4581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:47:21.145849 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142708 4581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:47:21.145849 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142710 4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:47:21.145849 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142713 4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:47:21.145849 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142715 4581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:47:21.145849 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142718 4581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:47:21.145849 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142720 4581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:47:21.145849 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142722 4581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:47:21.145849 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142725 4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:47:21.145849 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142727 4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:47:21.145849 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142730 4581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:47:21.146333 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142732 4581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:47:21.146333 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142737 4581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:47:21.146333 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142739 4581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:47:21.146333 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142742 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:47:21.146333 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142744 4581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:47:21.146333 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142747 4581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:47:21.146333 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142749 4581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:47:21.146333 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142752 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:47:21.146333 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142755 4581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:47:21.146333 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142757 4581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:47:21.146333 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142760 4581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:47:21.146333 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142762 4581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:47:21.146333 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142764 4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:47:21.146333 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142767 4581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:47:21.146333 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142769 4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:47:21.146333 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.142772 4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:47:21.146333 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143254 4581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:47:21.146333 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143260 4581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:47:21.146333 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143263 4581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:47:21.146333 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143266 4581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:47:21.146819 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143269 4581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:47:21.146819 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143271 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:47:21.146819 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143274 4581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:47:21.146819 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143277 4581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:47:21.146819 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143280 4581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:47:21.146819 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143282 4581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:47:21.146819 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143285 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:47:21.146819 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143287 4581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:47:21.146819 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143290 4581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:47:21.146819 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143292 4581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:47:21.146819 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143295 4581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:47:21.146819 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143298 4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:47:21.146819 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143300 4581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:47:21.146819 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143303 4581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:47:21.146819 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143306 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:47:21.146819 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143308 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:47:21.146819 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143311 4581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:47:21.146819 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143313 4581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:47:21.146819 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143316 4581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:47:21.147296 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143318 4581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:47:21.147296 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143321 4581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:47:21.147296 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143324 4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:47:21.147296 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143327 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:47:21.147296 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143329 4581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:47:21.147296 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143331 4581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:47:21.147296 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143335 4581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:47:21.147296 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143339 4581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:47:21.147296 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143342 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:47:21.147296 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143344 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:47:21.147296 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143347 4581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:47:21.147296 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143350 4581 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:47:21.147296 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143353 4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:47:21.147296 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143355 4581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:47:21.147296 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143358 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:47:21.147296 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143360 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:47:21.147296 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143362 4581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:47:21.147296 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143365 4581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:47:21.147296 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143367 4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:47:21.147750 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143370 4581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:47:21.147750 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143372 4581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:47:21.147750 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143375 4581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:47:21.147750 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143377 4581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:47:21.147750 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143380 4581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:47:21.147750 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143382 4581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:47:21.147750 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143385 4581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:47:21.147750 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143388 4581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:47:21.147750 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143390 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:47:21.147750 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143393 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:47:21.147750 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143396 4581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:47:21.147750 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143399 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:47:21.147750 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143402 4581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:47:21.147750 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143404 4581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:47:21.147750 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143407 4581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:47:21.147750 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143410 4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:47:21.147750 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143413 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:47:21.147750 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143416 4581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:47:21.147750 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143418 4581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:47:21.147750 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143421 4581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:47:21.148387 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143424 4581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:47:21.148387 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143426 4581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:47:21.148387 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143429 4581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:47:21.148387 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143432 4581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:47:21.148387 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143434 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:47:21.148387 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143437 4581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:47:21.148387 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143439 4581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:47:21.148387 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143442 4581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:47:21.148387 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143444 4581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:47:21.148387 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143446 4581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:47:21.148387 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143449 4581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:47:21.148387 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143451 4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:47:21.148387 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143455 4581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:47:21.148387 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143458 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:47:21.148387 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143461 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:47:21.148387 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143464 4581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:47:21.148387 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143466 4581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:47:21.148387 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143468 4581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:47:21.148387 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143471 4581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:47:21.149033 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143473 4581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:47:21.149033 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143476 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:47:21.149033 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143480 4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:47:21.149033 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143483 4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:47:21.149033 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143486 4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:47:21.149033 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143488 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:47:21.149033 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143491 4581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:47:21.149033 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143493 4581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:47:21.149033 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143496 4581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:47:21.149033 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143500 4581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:47:21.149033 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143502 4581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:47:21.149033 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143505 4581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:47:21.149033 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143507 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:47:21.149033 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143509 4581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:47:21.149033 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143512 4581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:47:21.149033 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143514 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:47:21.149033 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143517 4581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:47:21.149033 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143519 4581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:47:21.149033 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143522 4581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:47:21.149780 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143524 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:47:21.149780 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143526 4581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:47:21.149780 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143529 4581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:47:21.149780 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143531 4581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:47:21.149780 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143534 4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:47:21.149780 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.143536 4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:47:21.149780 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143634 4581 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 04:47:21.149780 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143641 4581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 04:47:21.149780 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143648 4581 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 04:47:21.149780 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143652 4581 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 04:47:21.149780 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143657 4581 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 04:47:21.149780 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143660 4581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 04:47:21.149780 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143664 4581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 04:47:21.149780 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143668 4581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 04:47:21.149780 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143671 4581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 04:47:21.149780 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143676 4581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 04:47:21.149780 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143679 4581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 04:47:21.149780 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143683 4581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 04:47:21.149780 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143686 4581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 04:47:21.149780 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143690 4581 flags.go:64] FLAG: --cgroup-root="" Nov 04 04:47:21.149780 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143693 4581 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 04:47:21.149780 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143696 4581 flags.go:64] FLAG: --client-ca-file="" Nov 04 04:47:21.149780 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143699 4581 flags.go:64] FLAG: --cloud-provider="external" Nov 04 04:47:21.150412 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143702 4581 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 04:47:21.150412 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143706 4581 flags.go:64] FLAG: --cluster-domain="" Nov 04 04:47:21.150412 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143709 4581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 04:47:21.150412 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143712 4581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 04:47:21.150412 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143714 4581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 04:47:21.150412 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143718 4581 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 04:47:21.150412 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143721 4581 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 04:47:21.150412 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143724 4581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 04:47:21.150412 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143727 4581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 04:47:21.150412 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143731 4581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 04:47:21.150412 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143734 4581 flags.go:64] FLAG: --contention-profiling="false" Nov 04 04:47:21.150412 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143737 4581 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 04:47:21.150412 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143740 4581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 04:47:21.150412 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143743 4581 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 04:47:21.150412 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143746 4581 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 04:47:21.150412 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143750 4581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 04:47:21.150412 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143753 4581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 04:47:21.150412 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143756 4581 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 04:47:21.150412 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143759 4581 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 04:47:21.150412 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143762 4581 flags.go:64] FLAG: --enable-server="true" Nov 04 04:47:21.150412 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143765 4581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 04:47:21.150412 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143769 4581 flags.go:64] FLAG: --event-burst="100" Nov 04 04:47:21.150412 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143772 4581 flags.go:64] FLAG: --event-qps="50" Nov 04 04:47:21.150412 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143775 4581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 04:47:21.151060 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143779 4581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 04:47:21.151060 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143782 4581 flags.go:64] FLAG: --eviction-hard="" Nov 04 04:47:21.151060 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143786 4581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 04:47:21.151060 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143789 4581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 04:47:21.151060 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143793 4581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 04:47:21.151060 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143796 4581 flags.go:64] FLAG: --eviction-soft="" Nov 04 04:47:21.151060 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143799 4581 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 04:47:21.151060 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143802 4581 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 04:47:21.151060 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143805 4581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 04:47:21.151060 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143808 4581 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 04:47:21.151060 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143811 4581 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 04:47:21.151060 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143814 4581 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 04:47:21.151060 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143817 4581 flags.go:64] FLAG: --feature-gates="" Nov 04 04:47:21.151060 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143821 4581 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 04:47:21.151060 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143824 4581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 04:47:21.151060 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143827 4581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 04:47:21.151060 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143830 4581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 04:47:21.151060 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143833 4581 flags.go:64] FLAG: --healthz-port="10248" Nov 04 04:47:21.151060 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143835 4581 flags.go:64] FLAG: --help="false" Nov 04 04:47:21.151060 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143838 4581 flags.go:64] FLAG: --hostname-override="ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:47:21.151060 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143841 4581 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 04:47:21.151060 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143844 4581 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 04:47:21.151060 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143847 4581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 04:47:21.151621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143850 4581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 04:47:21.151621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143854 4581 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 04:47:21.151621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143857 4581 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 04:47:21.151621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143859 4581 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 04:47:21.151621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143862 4581 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 04:47:21.151621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143865 4581 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 04:47:21.151621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143867 4581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 04:47:21.151621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143870 4581 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 04:47:21.151621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143873 4581 flags.go:64] FLAG: --kube-reserved="" Nov 04 04:47:21.151621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143878 4581 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 04:47:21.151621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143881 4581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 04:47:21.151621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143884 4581 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 04:47:21.151621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143886 4581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 04:47:21.151621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143891 4581 flags.go:64] FLAG: --lock-file="" Nov 04 04:47:21.151621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143894 4581 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 04:47:21.151621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143897 4581 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 04:47:21.151621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143899 4581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 04:47:21.151621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143905 4581 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 04:47:21.151621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143907 4581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 04:47:21.151621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143911 4581 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 04:47:21.151621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143914 4581 flags.go:64] FLAG: --logging-format="text" Nov 04 04:47:21.151621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143916 4581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 04:47:21.151621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143920 4581 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 04:47:21.151621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143923 4581 flags.go:64] FLAG: --manifest-url="" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143925 4581 flags.go:64] FLAG: --manifest-url-header="" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143930 4581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143933 4581 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143937 4581 flags.go:64] FLAG: --max-pods="110" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143940 4581 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143943 4581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143946 4581 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143949 4581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143952 4581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143955 4581 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143957 4581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143964 4581 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143967 4581 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143970 4581 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143973 4581 flags.go:64] FLAG: --pod-cidr="" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143976 4581 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.143978 4581 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144000 4581 flags.go:64] FLAG: --pods-per-core="0" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144004 4581 flags.go:64] FLAG: --port="10250" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144009 4581 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144012 4581 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-076b0bf1e744ee8f4" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144033 4581 flags.go:64] FLAG: --qos-reserved="" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144037 4581 flags.go:64] FLAG: --read-only-port="10255" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144041 4581 flags.go:64] FLAG: --register-node="true" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144044 4581 flags.go:64] FLAG: --register-with-taints="" Nov 04 04:47:21.152209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144048 4581 flags.go:64] FLAG: --registry-burst="10" Nov 04 04:47:21.152833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144051 4581 flags.go:64] FLAG: --registry-qps="5" Nov 04 04:47:21.152833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144053 4581 flags.go:64] FLAG: --reserved-cpus="" Nov 04 04:47:21.152833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144057 4581 flags.go:64] FLAG: --reserved-memory="" Nov 04 04:47:21.152833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144061 4581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 04:47:21.152833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144064 4581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 04:47:21.152833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144067 4581 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 04:47:21.152833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144069 4581 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 04:47:21.152833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144072 4581 flags.go:64] FLAG: --runonce="false" Nov 04 04:47:21.152833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144075 4581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 04:47:21.152833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144078 4581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 04:47:21.152833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144081 4581 flags.go:64] FLAG: --seccomp-default="false" Nov 04 04:47:21.152833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144084 4581 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 04:47:21.152833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144087 4581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 04:47:21.152833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144090 4581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 04:47:21.152833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144093 4581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 04:47:21.152833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144096 4581 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 04:47:21.152833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144099 4581 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 04:47:21.152833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144101 4581 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 04:47:21.152833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144104 4581 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 04:47:21.152833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144107 4581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 04:47:21.152833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144110 4581 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 04:47:21.152833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144113 4581 flags.go:64] FLAG: --system-cgroups="" Nov 04 04:47:21.152833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144115 4581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 04:47:21.152833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144122 4581 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 04:47:21.152833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144124 4581 flags.go:64] FLAG: --tls-cert-file="" Nov 04 04:47:21.153460 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144128 4581 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 04:47:21.153460 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144132 4581 flags.go:64] FLAG: --tls-min-version="" Nov 04 04:47:21.153460 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144136 4581 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 04:47:21.153460 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144138 4581 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 04:47:21.153460 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144141 4581 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 04:47:21.153460 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144144 4581 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 04:47:21.153460 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144147 4581 flags.go:64] FLAG: --v="2" Nov 04 04:47:21.153460 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144151 4581 flags.go:64] FLAG: --version="false" Nov 04 04:47:21.153460 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144155 4581 flags.go:64] FLAG: --vmodule="" Nov 04 04:47:21.153460 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144159 4581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 04:47:21.153460 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144163 4581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 04:47:21.153460 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144297 4581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:47:21.153460 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144301 4581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:47:21.153460 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144304 4581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:47:21.153460 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144307 4581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:47:21.153460 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144310 4581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:47:21.153460 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144312 4581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:47:21.153460 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144315 4581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:47:21.153460 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144317 4581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:47:21.153460 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144320 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:47:21.153460 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144322 4581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:47:21.153460 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144325 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:47:21.153460 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144327 4581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:47:21.154029 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144330 4581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:47:21.154029 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144333 4581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:47:21.154029 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144335 4581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:47:21.154029 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144337 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:47:21.154029 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144340 4581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:47:21.154029 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144343 4581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:47:21.154029 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144345 4581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:47:21.154029 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144349 4581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:47:21.154029 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144352 4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:47:21.154029 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144354 4581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:47:21.154029 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144356 4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:47:21.154029 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144360 4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:47:21.154029 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144362 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:47:21.154029 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144365 4581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:47:21.154029 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144367 4581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:47:21.154029 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144370 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:47:21.154029 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144372 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:47:21.154029 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144375 4581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:47:21.154029 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144379 4581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:47:21.154509 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144383 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:47:21.154509 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144389 4581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:47:21.154509 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144392 4581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:47:21.154509 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144394 4581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:47:21.154509 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144397 4581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:47:21.154509 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144399 4581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:47:21.154509 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144401 4581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:47:21.154509 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144404 4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:47:21.154509 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144406 4581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:47:21.154509 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144409 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:47:21.154509 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144411 4581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:47:21.154509 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144414 4581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:47:21.154509 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144416 4581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:47:21.154509 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144419 4581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:47:21.154509 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144421 4581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:47:21.154509 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144424 4581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:47:21.154509 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144427 4581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:47:21.154509 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144429 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:47:21.154509 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144432 4581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:47:21.154509 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144434 4581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:47:21.155116 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144438 4581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:47:21.155116 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144441 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:47:21.155116 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144444 4581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:47:21.155116 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144446 4581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:47:21.155116 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144451 4581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:47:21.155116 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144454 4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:47:21.155116 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144456 4581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:47:21.155116 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144459 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:47:21.155116 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144461 4581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:47:21.155116 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144464 4581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:47:21.155116 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144467 4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:47:21.155116 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144469 4581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:47:21.155116 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144472 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:47:21.155116 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144474 4581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:47:21.155116 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144477 4581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:47:21.155116 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144479 4581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:47:21.155116 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144482 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:47:21.155116 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144484 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:47:21.155116 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144487 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:47:21.155950 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144489 4581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:47:21.155950 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144492 4581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:47:21.155950 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144494 4581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:47:21.155950 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144497 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:47:21.155950 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144499 4581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:47:21.155950 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144501 4581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:47:21.155950 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144504 4581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:47:21.155950 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144506 4581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:47:21.155950 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144509 4581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:47:21.155950 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144511 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:47:21.155950 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144514 4581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:47:21.155950 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144516 4581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:47:21.155950 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144519 4581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:47:21.155950 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144522 4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:47:21.155950 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144524 4581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:47:21.155950 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144527 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:47:21.155950 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144529 4581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:47:21.155950 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144533 4581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:47:21.155950 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144535 4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:47:21.155950 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144538 4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:47:21.156511 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144540 4581 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:47:21.156511 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144543 4581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:47:21.156511 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144546 4581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:47:21.156511 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144548 4581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:47:21.156511 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144552 4581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:47:21.156511 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144556 4581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:47:21.156511 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144558 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:47:21.156511 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144561 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:47:21.156511 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144564 4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:47:21.156511 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144566 4581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:47:21.156511 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144569 4581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:47:21.156511 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144571 4581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:47:21.156511 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144574 4581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:47:21.156511 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144576 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:47:21.156511 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144579 4581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:47:21.156511 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144581 4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:47:21.156896 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.144607 4581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 04:47:21.156896 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144642 4581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:47:21.156896 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144646 4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:47:21.156896 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144649 4581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:47:21.156896 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144652 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:47:21.156896 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144655 4581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:47:21.156896 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144657 4581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:47:21.156896 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144660 4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:47:21.156896 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144662 4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:47:21.156896 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144665 4581 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:47:21.156896 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144667 4581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:47:21.156896 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144670 4581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:47:21.156896 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144673 4581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:47:21.156896 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144677 4581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:47:21.156896 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144682 4581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:47:21.157428 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144686 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:47:21.157428 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144689 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:47:21.157428 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144691 4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:47:21.157428 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144694 4581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:47:21.157428 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144696 4581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:47:21.157428 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144699 4581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:47:21.157428 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144702 4581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:47:21.157428 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144705 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:47:21.157428 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144707 4581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:47:21.157428 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144710 4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:47:21.157428 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144713 4581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:47:21.157428 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144716 4581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:47:21.157428 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144718 4581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:47:21.157428 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144721 4581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:47:21.157428 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144724 4581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:47:21.157428 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144727 4581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:47:21.157428 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144729 4581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:47:21.157428 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144732 4581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:47:21.157428 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144734 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:47:21.157428 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144737 4581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:47:21.158000 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144739 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:47:21.158000 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144742 4581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:47:21.158000 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144744 4581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:47:21.158000 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144746 4581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:47:21.158000 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144749 4581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:47:21.158000 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144752 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:47:21.158000 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144755 4581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:47:21.158000 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144758 4581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:47:21.158000 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144760 4581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:47:21.158000 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144763 4581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:47:21.158000 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144765 4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:47:21.158000 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144769 4581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:47:21.158000 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144771 4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:47:21.158000 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144774 4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:47:21.158000 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144776 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:47:21.158000 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144779 4581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:47:21.158000 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144781 4581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:47:21.158000 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144784 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:47:21.158000 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144786 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:47:21.158000 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144789 4581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:47:21.158506 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144793 4581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:47:21.158506 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144797 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:47:21.158506 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144799 4581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:47:21.158506 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144802 4581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:47:21.158506 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144805 4581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:47:21.158506 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144808 4581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:47:21.158506 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144810 4581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:47:21.158506 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144812 4581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:47:21.158506 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144815 4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:47:21.158506 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144817 4581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:47:21.158506 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144820 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:47:21.158506 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144823 4581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:47:21.158506 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144825 4581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:47:21.158506 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144827 4581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:47:21.158506 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144830 4581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:47:21.158506 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144832 4581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:47:21.158506 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144835 4581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:47:21.158506 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144838 4581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:47:21.158506 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144840 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:47:21.158965 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144843 4581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:47:21.158965 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144845 4581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:47:21.158965 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144848 4581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:47:21.158965 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144851 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:47:21.158965 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144855 4581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:47:21.158965 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144857 4581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:47:21.158965 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144860 4581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:47:21.158965 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144862 4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:47:21.158965 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144865 4581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:47:21.158965 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144867 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:47:21.158965 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144870 4581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:47:21.158965 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144873 4581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:47:21.158965 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144876 4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:47:21.158965 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144878 4581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:47:21.158965 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144881 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:47:21.158965 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144883 4581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:47:21.158965 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144886 4581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:47:21.158965 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144889 4581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:47:21.158965 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144891 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:47:21.159434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144893 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:47:21.159434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144896 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:47:21.159434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144898 4581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:47:21.159434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144901 4581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:47:21.159434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144903 4581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:47:21.159434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144906 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:47:21.159434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144908 4581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:47:21.159434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144911 4581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:47:21.159434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144914 4581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:47:21.159434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144916 4581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:47:21.159434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144918 4581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:47:21.159434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144921 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:47:21.159434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144923 4581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:47:21.159434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.144925 4581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:47:21.159434 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.149165 4581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 04:47:21.159434 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.149195 4581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 04:47:21.159434 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.149222 4581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 04:47:21.159434 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.149229 4581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 04:47:21.159434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149408 4581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:47:21.159434 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149415 4581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:47:21.159915 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149425 4581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:47:21.159915 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149429 4581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:47:21.159915 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149434 4581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:47:21.159915 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149438 4581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:47:21.159915 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149442 4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:47:21.159915 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149447 4581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:47:21.159915 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149451 4581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:47:21.159915 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149455 4581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:47:21.159915 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149460 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:47:21.159915 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149464 4581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:47:21.159915 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149468 4581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:47:21.159915 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149472 4581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:47:21.159915 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149482 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:47:21.159915 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149487 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:47:21.159915 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149491 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:47:21.159915 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149496 4581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:47:21.159915 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149500 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:47:21.159915 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149505 4581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:47:21.159915 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149509 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:47:21.160384 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149514 4581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:47:21.160384 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149518 4581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:47:21.160384 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149523 4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:47:21.160384 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149528 4581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:47:21.160384 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149532 4581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:47:21.160384 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149537 4581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:47:21.160384 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149545 4581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:47:21.160384 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149550 4581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:47:21.160384 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149555 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:47:21.160384 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149559 4581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:47:21.160384 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149564 4581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:47:21.160384 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149568 4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:47:21.160384 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149572 4581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:47:21.160384 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149577 4581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:47:21.160384 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149581 4581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:47:21.160384 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149585 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:47:21.160384 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149589 4581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:47:21.160384 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149593 4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:47:21.160384 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149603 4581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:47:21.160384 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149607 4581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:47:21.160873 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149612 4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:47:21.160873 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149616 4581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:47:21.160873 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149620 4581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:47:21.160873 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149624 4581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:47:21.160873 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149628 4581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:47:21.160873 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149632 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:47:21.160873 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149636 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:47:21.160873 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149642 4581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:47:21.160873 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149715 4581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:47:21.160873 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149723 4581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:47:21.160873 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149730 4581 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:47:21.160873 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149735 4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:47:21.160873 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149741 4581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:47:21.160873 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149746 4581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:47:21.160873 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149751 4581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:47:21.160873 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149755 4581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:47:21.160873 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149762 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:47:21.160873 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149811 4581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:47:21.160873 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149946 4581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:47:21.160873 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149953 4581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:47:21.161504 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149958 4581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:47:21.161504 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149963 4581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:47:21.161504 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149970 4581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:47:21.161504 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149973 4581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:47:21.161504 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149977 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:47:21.161504 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.149979 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:47:21.161504 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150002 4581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:47:21.161504 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150005 4581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:47:21.161504 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150008 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:47:21.161504 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150011 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:47:21.161504 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150014 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:47:21.161504 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150021 4581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:47:21.161504 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150024 4581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:47:21.161504 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150026 4581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:47:21.161504 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150029 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:47:21.161504 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150031 4581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:47:21.161504 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150034 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:47:21.161504 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150037 4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:47:21.161504 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150040 4581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:47:21.162024 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150042 4581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:47:21.162024 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150045 4581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:47:21.162024 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150047 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:47:21.162024 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150050 4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:47:21.162024 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150053 4581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:47:21.162024 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150056 4581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:47:21.162024 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150059 4581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:47:21.162024 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150062 4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:47:21.162024 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150064 4581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:47:21.162024 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150066 4581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:47:21.162024 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150069 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:47:21.162024 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150072 4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:47:21.162024 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150074 4581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:47:21.162024 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150078 4581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:47:21.162024 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150080 4581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:47:21.162024 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150082 4581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:47:21.162024 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150086 4581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:47:21.162024 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150088 4581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:47:21.162024 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150091 4581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:47:21.162024 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150094 4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:47:21.162513 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150096 4581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:47:21.162513 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150099 4581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:47:21.162513 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150103 4581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:47:21.162513 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150106 4581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:47:21.162513 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150108 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:47:21.162513 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150111 4581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:47:21.162513 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.150154 4581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 04:47:21.162513 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150300 4581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:47:21.162513 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150305 4581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:47:21.162513 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150309 4581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:47:21.162513 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150312 4581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:47:21.162513 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150315 4581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:47:21.162513 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150317 4581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:47:21.162513 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150320 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:47:21.162513 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150323 4581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:47:21.162513 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150326 4581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:47:21.162922 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150328 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:47:21.162922 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150331 4581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:47:21.162922 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150333 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:47:21.162922 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150336 4581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:47:21.162922 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150339 4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:47:21.162922 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150341 4581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:47:21.162922 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150344 4581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:47:21.162922 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150347 4581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:47:21.162922 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150351 4581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:47:21.162922 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150354 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:47:21.162922 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150357 4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:47:21.162922 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150360 4581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:47:21.162922 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150363 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:47:21.162922 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150366 4581 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:47:21.162922 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150369 4581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:47:21.162922 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150372 4581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:47:21.162922 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150374 4581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:47:21.162922 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150377 4581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:47:21.162922 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150379 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:47:21.163444 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150382 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:47:21.163444 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150384 4581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:47:21.163444 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150386 4581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:47:21.163444 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150389 4581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:47:21.163444 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150391 4581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:47:21.163444 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150394 4581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:47:21.163444 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150396 4581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:47:21.163444 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150399 4581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:47:21.163444 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150401 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:47:21.163444 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150404 4581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:47:21.163444 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150406 4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:47:21.163444 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150409 4581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:47:21.163444 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150412 4581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:47:21.163444 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150414 4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:47:21.163444 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150417 4581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:47:21.163444 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150420 4581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:47:21.163444 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150423 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:47:21.163444 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150427 4581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:47:21.163444 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150429 4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:47:21.163444 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150432 4581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:47:21.163931 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150434 4581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:47:21.163931 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150437 4581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:47:21.163931 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150440 4581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:47:21.163931 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150442 4581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:47:21.163931 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150445 4581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:47:21.163931 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150447 4581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:47:21.163931 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150451 4581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:47:21.163931 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150454 4581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:47:21.163931 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150458 4581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:47:21.163931 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150460 4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:47:21.163931 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150463 4581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:47:21.163931 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150465 4581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:47:21.163931 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150468 4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:47:21.163931 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150470 4581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:47:21.163931 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150472 4581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:47:21.163931 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150475 4581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:47:21.163931 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150477 4581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:47:21.163931 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150479 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:47:21.163931 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150482 4581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:47:21.164411 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150484 4581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:47:21.164411 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150487 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:47:21.164411 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150489 4581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:47:21.164411 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150491 4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:47:21.164411 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150495 4581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:47:21.164411 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150497 4581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:47:21.164411 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150500 4581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:47:21.164411 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150502 4581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:47:21.164411 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150505 4581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:47:21.164411 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150507 4581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:47:21.164411 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150511 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:47:21.164411 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150513 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:47:21.164411 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150516 4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:47:21.164411 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150518 4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:47:21.164411 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150521 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:47:21.164411 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150523 4581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:47:21.164411 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150526 4581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:47:21.164411 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150528 4581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:47:21.164411 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150531 4581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:47:21.164928 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150533 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:47:21.164928 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150536 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:47:21.164928 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150538 4581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:47:21.164928 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150541 4581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:47:21.164928 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150544 4581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:47:21.164928 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150546 4581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:47:21.164928 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150549 4581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:47:21.164928 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150551 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:47:21.164928 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150554 4581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:47:21.164928 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150556 4581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:47:21.164928 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150558 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:47:21.164928 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150561 4581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:47:21.164928 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150563 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:47:21.164928 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150565 4581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:47:21.164928 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150568 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:47:21.164928 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150570 4581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:47:21.164928 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150573 4581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:47:21.164928 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150575 4581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:47:21.164928 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150578 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:47:21.165396 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150581 4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:47:21.165396 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.150604 4581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 04:47:21.165396 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150637 4581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:47:21.165396 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150641 4581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:47:21.165396 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150645 4581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:47:21.165396 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150648 4581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:47:21.165396 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150650 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:47:21.165396 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150653 4581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:47:21.165396 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150656 4581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:47:21.165396 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150658 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:47:21.165396 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150661 4581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:47:21.165396 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150663 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:47:21.165396 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150666 4581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:47:21.165396 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150668 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:47:21.165396 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150670 4581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:47:21.165396 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150673 4581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:47:21.165825 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150675 4581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:47:21.165825 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150678 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:47:21.165825 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150680 4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:47:21.165825 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150683 4581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:47:21.165825 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150685 4581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:47:21.165825 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150688 4581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:47:21.165825 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150690 4581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:47:21.165825 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150693 4581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:47:21.165825 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150696 4581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:47:21.165825 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150698 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:47:21.165825 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150700 4581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:47:21.165825 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150703 4581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:47:21.165825 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150706 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:47:21.165825 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150708 4581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:47:21.165825 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150711 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:47:21.165825 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150713 4581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:47:21.165825 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150717 4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:47:21.165825 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150719 4581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:47:21.165825 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150722 4581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:47:21.166493 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150724 4581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:47:21.166493 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150729 4581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:47:21.166493 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150732 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:47:21.166493 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150735 4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:47:21.166493 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150739 4581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:47:21.166493 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150741 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:47:21.166493 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150744 4581 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:47:21.166493 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150747 4581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:47:21.166493 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150749 4581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:47:21.166493 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150752 4581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:47:21.166493 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150754 4581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:47:21.166493 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150756 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:47:21.166493 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150759 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:47:21.166493 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150761 4581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:47:21.166493 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150764 4581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:47:21.166493 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150766 4581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:47:21.166493 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150768 4581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:47:21.166493 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150771 4581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:47:21.166493 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150773 4581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:47:21.166493 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150776 4581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:47:21.167272 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150784 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:47:21.167272 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150787 4581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:47:21.167272 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150789 4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:47:21.167272 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150791 4581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:47:21.167272 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150794 4581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:47:21.167272 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150797 4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:47:21.167272 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150799 4581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:47:21.167272 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150801 4581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:47:21.167272 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150804 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:47:21.167272 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150807 4581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:47:21.167272 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150810 4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:47:21.167272 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150813 4581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:47:21.167272 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150816 4581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:47:21.167272 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150818 4581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:47:21.167272 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150821 4581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:47:21.167272 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150823 4581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:47:21.167272 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150826 4581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:47:21.167272 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150828 4581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:47:21.167272 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150833 4581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:47:21.167739 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150837 4581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:47:21.167739 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150839 4581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:47:21.167739 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150842 4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:47:21.167739 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150845 4581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:47:21.167739 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150847 4581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:47:21.167739 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150849 4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:47:21.167739 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150852 4581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:47:21.167739 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150854 4581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:47:21.167739 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150856 4581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:47:21.167739 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150859 4581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:47:21.167739 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150861 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:47:21.167739 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150864 4581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:47:21.167739 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150866 4581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:47:21.167739 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150868 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:47:21.167739 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150871 4581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:47:21.167739 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150873 4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:47:21.167739 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150875 4581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:47:21.167739 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150878 4581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:47:21.167739 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150880 4581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:47:21.167739 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150883 4581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:47:21.168335 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150887 4581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:47:21.168335 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150889 4581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:47:21.168335 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150892 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:47:21.168335 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150894 4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:47:21.168335 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150897 4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:47:21.168335 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150900 4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:47:21.168335 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150902 4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:47:21.168335 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150905 4581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:47:21.168335 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150907 4581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:47:21.168335 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150910 4581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:47:21.168335 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150912 4581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:47:21.168335 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150915 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:47:21.168335 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150917 4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:47:21.168335 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:21.150920 4581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:47:21.168335 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.150978 4581 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 04:47:21.168335 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.152011 4581 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 04 04:47:21.168335 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.152076 4581 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 04 04:47:21.168335 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.152263 4581 server.go:1014] "Starting client certificate rotation" Nov 04 04:47:21.169020 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.152393 4581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 04:47:21.169020 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.152431 4581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 04:41:41 +0000 UTC" deadline="2028-07-12 09:19:32.203271074 +0000 UTC" Nov 04 04:47:21.169020 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.152468 4581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14788h32m11.050805101s" Nov 04 04:47:21.169020 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.152742 4581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 04:47:21.169020 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.153896 4581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 04:47:21.169020 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.155454 4581 log.go:25] "Validated CRI v1 runtime API" Nov 04 04:47:21.169020 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.157080 4581 log.go:25] "Validated CRI v1 image API" Nov 04 04:47:21.169020 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.157901 4581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 04:47:21.169020 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.159435 4581 fs.go:135] Filesystem UUIDs: map[0be9cc94-3840-4d67-9174-653735a4dee2:/dev/nvme0n1p3 436e5e68-db6c-489d-831d-97a4a007cecb:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 04 04:47:21.169318 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.159455 4581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1877e32e770417de592c93724586fd08b55c01287cdef0826be75b746840cc31/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1877e32e770417de592c93724586fd08b55c01287cdef0826be75b746840cc31/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b783f131c1ac45c5c286605c31b12d834ba7248aed8d4c244a1e791fdd5beff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b783f131c1ac45c5c286605c31b12d834ba7248aed8d4c244a1e791fdd5beff/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d83e82749c5a46c027df325004fd629f95cf83c9a4b282a3fe605c67574425f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d83e82749c5a46c027df325004fd629f95cf83c9a4b282a3fe605c67574425f/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c1e06c23ceafefa52bd98082bbab3815560178a5df8c33c4d1d034e70ea9f5e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c1e06c23ceafefa52bd98082bbab3815560178a5df8c33c4d1d034e70ea9f5e/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6fcdfae34ced4daf239c88b5da340350a314a358d75057c991e797ee9d5af075/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6fcdfae34ced4daf239c88b5da340350a314a358d75057c991e797ee9d5af075/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9fba49fabd21e22b7aa117c7a67b47816b1f2967db50873193f3b1b6b35e4fc3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9fba49fabd21e22b7aa117c7a67b47816b1f2967db50873193f3b1b6b35e4fc3/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aebd3899e7827a79b9a2d9833edc445155cf66b5d9c1f5f58aebf985c564bf6c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aebd3899e7827a79b9a2d9833edc445155cf66b5d9c1f5f58aebf985c564bf6c/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb0bf32ff56b9b965f8693e29ac897a5a843a5597237f7458aa5a5db509f71d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb0bf32ff56b9b965f8693e29ac897a5a843a5597237f7458aa5a5db509f71d3/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bdf831998fa83bbf9a15393773f1db7c81190256b00a0f1ceeff03dc4451b05c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bdf831998fa83bbf9a15393773f1db7c81190256b00a0f1ceeff03dc4451b05c/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4b7c4e9635fe7d938e45641be9f043fd487a93b37ebac50bacca07a81b1ba51/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4b7c4e9635fe7d938e45641be9f043fd487a93b37ebac50bacca07a81b1ba51/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c60f1c25de9e2eb62b826c1324da16737ba57c99e55c4ed0ff2044571e318911/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c60f1c25de9e2eb62b826c1324da16737ba57c99e55c4ed0ff2044571e318911/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c9abbebf0d2223426b71b38556e7fd55a7058957a43cc318c0e6309ba72d5c46/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c9abbebf0d2223426b71b38556e7fd55a7058957a43cc318c0e6309ba72d5c46/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cd738a1ec5d9f08c2efc725dd095c882117ba37f285517c37650b400964c93d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cd738a1ec5d9f08c2efc725dd095c882117ba37f285517c37650b400964c93d4/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb064591ea1b60ade7aeb414a546736585ead71970e8ca8005e7e75d9b225e72/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb064591ea1b60ade7aeb414a546736585ead71970e8ca8005e7e75d9b225e72/userdata/shm major:0 minor:262 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f3e092af629e4a957bf5377e9721b5f7a1302fdbdb8feba985b3f5fe090cca6d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f3e092af629e4a957bf5377e9721b5f7a1302fdbdb8feba985b3f5fe090cca6d/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb0861f4193124e2573bd818c65ccf7f2bb0bd9e1a74049ad4341c293ad48882/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb0861f4193124e2573bd818c65ccf7f2bb0bd9e1a74049ad4341c293ad48882/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/030a8f10-c4b1-45b2-abb8-88ed48e53a80/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/030a8f10-c4b1-45b2-abb8-88ed48e53a80/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12f1a157-ce5b-4040-98de-ed53507268db/volumes/kubernetes.io~projected/kube-api-access-98rdv:{mountpoint:/var/lib/kubelet/pods/12f1a157-ce5b-4040-98de-ed53507268db/volumes/kubernetes.io~projected/kube-api-access-98rdv major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12f1a157-ce5b-4040-98de-ed53507268db/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/12f1a157-ce5b-4040-98de-ed53507268db/volumes/kubernetes.io~secret/metrics-certs major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24bc4691-e47f-4f98-ad47-f14654d95d28/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/24bc4691-e47f-4f98-ad47-f14654d95d28/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26237d52-9119-41f2-919a-5e5153ea223a/volumes/kubernetes.io~projected/kube-api-access-nd2j4:{mountpoint:/var/lib/kubelet/pods/26237d52-9119-41f2-919a-5e5153ea223a/volumes/kubernetes.io~projected/kube-api-access-nd2j4 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27a76981-142e-4959-aac0-daaffc4f9b95/volumes/kubernetes.io~projected/kube-api-access-xxwbx:{mountpoint:/var/lib/kubelet/pods/27a76981-142e-4959-aac0-daaffc4f9b95/volumes/kubernetes.io~projected/kube-api-access-xxwbx major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bb73603-5b6f-4a77-a39f-033b1f80059f/volumes/kubernetes.io~projected/kube-api-access-sjpz9:{mountpoint:/var/lib/kubelet/pods/3bb73603-5b6f-4a77-a39f-033b1f80059f/volumes/kubernetes.io~projected/kube-api-access-sjpz9 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bb73603-5b6f-4a77-a39f-033b1f80059f/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/3bb73603-5b6f-4a77-a39f-033b1f80059f/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/565b853c-b917-41a5-b739-368db09dac1f/volumes/kubernetes.io~projected/kube-api-access-m9g5g:{mountpoint:/var/lib/kubelet/pods/565b853c-b917-41a5-b739-368db09dac1f/volumes/kubernetes.io~projected/kube-api-access-m9g5g major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63e7d159-5f5f-4802-997b-5bfc47f1dad2/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/63e7d159-5f5f-4802-997b-5bfc47f1dad2/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63e7d159-5f5f-4802-997b-5bfc47f1dad2/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/63e7d159-5f5f-4802-997b-5bfc47f1dad2/volumes/kubernetes.io~empty-dir/tmp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63e7d159-5f5f-4802-997b-5bfc47f1dad2/volumes/kubernetes.io~projected/kube-api-access-2jj28:{mountpoint:/var/lib/kubelet/pods/63e7d159-5f5f-4802-997b-5bfc47f1dad2/volumes/kubernetes.io~projected/kube-api-access-2jj28 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64661f31-04d2-41e7-aab7-28000a68fc64/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/64661f31-04d2-41e7-aab7-28000a68fc64/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64661f31-04d2-41e7-aab7-28000a68fc64/volumes/kubernetes.io~projected/kube-api-access-q8q7s:{mountpoint:/var/lib/kubelet/pods/64661f31-04d2-41e7-aab7-28000a68fc64/volumes/kubernetes.io~projected/kube-api-access-q8q7s major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64661f31-04d2-41e7-aab7-28000a68fc64/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/64661f31-04d2-41e7-aab7-28000a68fc64/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b/volumes/kubernetes.io~projected/kube-api-access-jjmwq:{mountpoint:/var/lib/kubelet/pods/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b/volumes/kubernetes.io~projected/kube-api-access-jjmwq major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/acddcc2f-2b31-478b-a0de-552929cd7bd4/volumes/kubernetes.io~projected/kube-api-access-rjj9c:{mountpoint:/var/lib/kubelet/pods/acddcc2f-2b31-478b-a0de-552929cd7bd4/volumes/kubernetes.io~projected/kube-api-access-rjj9c major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3e3cff9-0366-457e-a337-32722ca38e00/volumes/kubernetes.io~projected/kube-api-access-p5s2h:{mountpoint:/var/lib/kubelet/pods/c3e3cff9-0366-457e-a337-32722ca38e00/volumes/kubernetes.io~projected/kube-api-access-p5s2h major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3e3cff9-0366-457e-a337-32722ca38e00/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/c3e3cff9-0366-457e-a337-32722ca38e00/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0e2510f-0579-4594-bd3b-a3dadf3fb81a/volumes/kubernetes.io~projected/kube-api-access-ct442:{mountpoint:/var/lib/kubelet/pods/d0e2510f-0579-4594-bd3b-a3dadf3fb81a/volumes/kubernetes.io~projected/kube-api-access-ct442 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e407d116-de8e-4a31-bcee-88546ae61f57/volumes/kubernetes.io~projected/kube-api-access-s9rtr:{mountpoint:/var/lib/kubelet/pods/e407d116-de8e-4a31-bcee-88546ae61f57/volumes/kubernetes.io~projected/kube-api-access-s9rtr major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e407d116-de8e-4a31-bcee-88546ae61f57/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/e407d116-de8e-4a31-bcee-88546ae61f57/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed8f18be-cea2-4c8f-a143-25ddd48b1a74/volumes/kubernetes.io~projected/kube-api-access-w87hs:{mountpoint:/var/lib/kubelet/pods/ed8f18be-cea2-4c8f-a143-25ddd48b1a74/volumes/kubernetes.io~projected/kube-api-access-w87hs major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f5b27917df3763c0ff6bb36c90fa408e8b8d5f90f4fc5391a70ee89825e9fd2c/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/9e6aeb44bca1203261ca80579448d5f611a4c5b9b361084910729f5faf75e7f9/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/bc4b21d50055a08a870ef77fef755b9139133eba7edda099a04a36f9bba82bd9/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/866aac08986b2a29cbeaaf456dd2c7884a09eda63e8b7080cadde7a631b1566a/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/45d73857d4470390bc04743c11e10d24a2d248e359db160a2c4bef2d37affbbb/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/b7d79f03d875c9f21dfc8c6a02e8ff53ae7e68704f17e5a4b54bb1a9130e53ca/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/379d6334d80dfca04276fb9e1cb2c43799c9192236a1b1f164b8d9c675cad0e2/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/0ec8e5dcf799b7a8bd807ca904e58283df6b3f14e51a0a430ccde0cfa5361803/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/574773eaeb15be23b5569fcd7fcdd1f788f05548c8aa1d2cd24fdbf745a4459b/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/39bb5cc7d0148fc23d21f0d4bb9b87b1d5c1be33a44ee21896387af136d7170b/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/55d0ec7f8a9bb805c419cfffa0fe2a708ab48bf709966e7c8a2b98b34407ed60/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/3730bf48a4d8a43bb8425d394256be2e3aa80239cfe704ad8895f2e664836ddd/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a6c1e80cc889c099a9947ce5e812acacdc406a4368770d191f1685bfe11f7c99/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/fbab42790c60074c14dc2c284a23741f1126226bbc839b66ede1faeed57e7d3f/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/c25d3ade29422a48a23552541534bed3cebcfcaca7563c2f6f63480bf55d4aa0/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/b2de97271c08aa84225fc4fb4fa5e893338982ac34cd5b0778612d91482a44ef/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/ea166552167e9a10f15044ee694191c1af5dbd560db6c42c130f300b8fcc309a/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-176:{mountpoint:/var/lib/containers/storage/overlay/7a128727b12b691de84827181dbeacc139a84a78ac7d276806971eaa1adbacab/merged major:0 minor:176 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/398e3c3cd171b83c2b7ce9205e8f9a3efd6f5643630398675a1782532ff007af/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/c9d5d4ee671c2164934f2b1f438147fdf1f3a4ad6bdd55aff249b44896816c9c/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f1532dc7f791a6dc7ae90f9260c06970fe367aac56ec970595b0c02ab49e97d5/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-208:{mountpoint:/var/lib/containers/storage/overlay/d07afc6ec73aac6e741c12ce3dc1898b810de70c60d6855465b2f853e3281786/merged major:0 minor:208 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/cd4ced525a818992e9dda7b3e1b11a84758c2093ced703c3ea79f4cf3d60d7e2/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/eb678f2bb8ab9d71d0147db2c143a763b08db387edff5b9263cb9b81e63cb6bf/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/d06980bb35cc845b89071e70af0490bef72a68c4bff21898a777dc523a573863/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/8c0b50a001f30490917f08c37b47bfe95750033edabb988edd10c4b4cab47e65/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/a990d5fdc4b38d9c817bbf0bad166cc8dfc7b03db932df1f8cf256a5cf591c79/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/fc44bf6e7bd60e8cb63399ae21abb872a270df8cafaf152d7d57847639c75b97/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/df5d63911f01ac50ba05a793df350a2b832789bf27bcd0627cb0290cb1cea59b/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/0ced71b90eda3f1dea3f41aa9b11b738e67ebd19446cc350874183683c4ac3b8/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/093d4f2a6b5f864a92c1fa141a3a539cda7527b6e2b7cc40925b0fc988242bf8/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/3675b88148977662a33d618baef90783cbe11e5f595335b08570b28a6fb88a2f/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/8209f2afa3b612912fa6b3b2249476a8c22dc040b7d87e84d4228a3caac93add/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/37ea12dfb9aeeb56fc1e520abd10bf18600d1864db9d7f0fe1b702b4536cf37e/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/9e4ad7471e53fa8fa6fd0abaf0530f936688b082497bfa86265743ed5cb055b5/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/199560a1710c3bac7703499d0ed2971b76f7b9bdb936224f8894b521fae73c12/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/5de7cb5398050b7c1148d43b0c37012bfc406b49feb8a7d188b5d69ecd53f898/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/ec655dc756fd9c135f8542f4aaf761f3823392ab1070c2a6b8e635869f279b41/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/fd6ac82f644c27ec4be6de8b25e3cee792cbf0166b36eadb5218d0776c3aa4d4/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/66ec6a514837a0dce2fb2207aff43d380d34c8cd8eb3e58bff242bfea426ff09/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/731892e9a7ee65461fd3d0be4fdf81b2a05de04fbce4ddb6dc4fe8e2cba61da7/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/690d5ed6288121be33e89338b7c94b2b9efa1b4ae93f4ea620a3852be8e19bbd/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/39e374258d61bc1aeca4b34a4b961f4aebb79eb7e57863b9afeb730c54e7863f/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/0ef85d4530961ab358ef6a084a71b4bfcc6de03434a3e343aeb3418f5c60e0c8/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/9c31ac0cc6dfd220ab3c921115fc22bf53e7815472ff2398454eb47572fcbae3/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 04 04:47:21.169479 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.165749 4581 manager.go:217] Machine: {Timestamp:2026-11-04 04:47:21.165165838 +0000 UTC m=+0.056022848 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091334 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22db75a07c4752562af06c9ea31ad4 SystemUUID:ec22db75-a07c-4752-562a-f06c9ea31ad4 BootID:afa9e075-59f4-4b5f-9c73-cb4c19e08c48 Filesystems:[{Device:/run/containers/storage/overlay-containers/eb064591ea1b60ade7aeb414a546736585ead71970e8ca8005e7e75d9b225e72/userdata/shm DeviceMajor:0 DeviceMinor:262 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/63e7d159-5f5f-4802-997b-5bfc47f1dad2/volumes/kubernetes.io~projected/kube-api-access-2jj28 DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e407d116-de8e-4a31-bcee-88546ae61f57/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:257 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/63e7d159-5f5f-4802-997b-5bfc47f1dad2/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b783f131c1ac45c5c286605c31b12d834ba7248aed8d4c244a1e791fdd5beff/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4b7c4e9635fe7d938e45641be9f043fd487a93b37ebac50bacca07a81b1ba51/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/26237d52-9119-41f2-919a-5e5153ea223a/volumes/kubernetes.io~projected/kube-api-access-nd2j4 DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/27a76981-142e-4959-aac0-daaffc4f9b95/volumes/kubernetes.io~projected/kube-api-access-xxwbx DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9fba49fabd21e22b7aa117c7a67b47816b1f2967db50873193f3b1b6b35e4fc3/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/24bc4691-e47f-4f98-ad47-f14654d95d28/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d0e2510f-0579-4594-bd3b-a3dadf3fb81a/volumes/kubernetes.io~projected/kube-api-access-ct442 DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c60f1c25de9e2eb62b826c1324da16737ba57c99e55c4ed0ff2044571e318911/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cd738a1ec5d9f08c2efc725dd095c882117ba37f285517c37650b400964c93d4/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb0861f4193124e2573bd818c65ccf7f2bb0bd9e1a74049ad4341c293ad48882/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1877e32e770417de592c93724586fd08b55c01287cdef0826be75b746840cc31/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f3e092af629e4a957bf5377e9721b5f7a1302fdbdb8feba985b3f5fe090cca6d/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/64661f31-04d2-41e7-aab7-28000a68fc64/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/acddcc2f-2b31-478b-a0de-552929cd7bd4/volumes/kubernetes.io~projected/kube-api-access-rjj9c DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/565b853c-b917-41a5-b739-368db09dac1f/volumes/kubernetes.io~projected/kube-api-access-m9g5g DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c9abbebf0d2223426b71b38556e7fd55a7058957a43cc318c0e6309ba72d5c46/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb0bf32ff56b9b965f8693e29ac897a5a843a5597237f7458aa5a5db509f71d3/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/c3e3cff9-0366-457e-a337-32722ca38e00/volumes/kubernetes.io~projected/kube-api-access-p5s2h DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bdf831998fa83bbf9a15393773f1db7c81190256b00a0f1ceeff03dc4451b05c/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aebd3899e7827a79b9a2d9833edc445155cf66b5d9c1f5f58aebf985c564bf6c/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/030a8f10-c4b1-45b2-abb8-88ed48e53a80/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/12f1a157-ce5b-4040-98de-ed53507268db/volumes/kubernetes.io~projected/kube-api-access-98rdv DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-208 DeviceMajor:0 DeviceMinor:208 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12f1a157-ce5b-4040-98de-ed53507268db/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6fcdfae34ced4daf239c88b5da340350a314a358d75057c991e797ee9d5af075/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b/volumes/kubernetes.io~projected/kube-api-access-jjmwq DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ed8f18be-cea2-4c8f-a143-25ddd48b1a74/volumes/kubernetes.io~projected/kube-api-access-w87hs DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/64661f31-04d2-41e7-aab7-28000a68fc64/volumes/kubernetes.io~projected/kube-api-access-q8q7s DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-176 DeviceMajor:0 DeviceMinor:176 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3bb73603-5b6f-4a77-a39f-033b1f80059f/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3bb73603-5b6f-4a77-a39f-033b1f80059f/volumes/kubernetes.io~projected/kube-api-access-sjpz9 DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/63e7d159-5f5f-4802-997b-5bfc47f1dad2/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d83e82749c5a46c027df325004fd629f95cf83c9a4b282a3fe605c67574425f/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/64661f31-04d2-41e7-aab7-28000a68fc64/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/c3e3cff9-0366-457e-a337-32722ca38e00/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e407d116-de8e-4a31-bcee-88546ae61f57/volumes/kubernetes.io~projected/kube-api-access-s9rtr DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c1e06c23ceafefa52bd98082bbab3815560178a5df8c33c4d1d034e70ea9f5e/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1877e32e770417d MacAddress:22:78:f5:3f:bc:a7 Speed:10000 Mtu:8901} {Name:6fcdfae34ced4da MacAddress:e2:23:09:a7:d3:1e Speed:10000 Mtu:8901} {Name:aebd3899e7827a7 MacAddress:e2:7a:30:88:8a:69 Speed:10000 Mtu:8901} {Name:bb0bf32ff56b9b9 MacAddress:3a:77:34:a9:b6:97 Speed:10000 Mtu:8901} {Name:bdf831998fa83bb MacAddress:6a:55:d3:61:fb:bb Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:5a:67:1b:8e:a3 Speed:0 Mtu:9001} {Name:br-int MacAddress:62:8e:9f:67:0a:0c Speed:0 Mtu:8901} {Name:eb064591ea1b60a MacAddress:66:28:f9:ed:9c:c3 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:5a:67:1b:8e:a3 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:9e:6d:f1:44:5d:9b Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:26:f0:dd:d4:b9:45 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 04:47:21.169479 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.166418 4581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 04:47:21.169479 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.166502 4581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 04:47:21.169479 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.166867 4581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 04:47:21.169479 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.166904 4581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-132.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 04:47:21.169479 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.167548 4581 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 04:47:21.169479 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.167559 4581 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 04:47:21.169479 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.167570 4581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 04:47:21.169479 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.167590 4581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 04:47:21.169479 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.167685 4581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 04:47:21.169479 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.167852 4581 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 04:47:21.169479 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.167953 4581 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 04:47:21.169479 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.168008 4581 kubelet.go:493] "Attempting to sync node with API server" Nov 04 04:47:21.169479 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.168020 4581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 04:47:21.169479 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.168030 4581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 04:47:21.169479 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.168038 4581 kubelet.go:405] "Adding apiserver pod source" Nov 04 04:47:21.169479 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.168046 4581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 04:47:21.169479 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.168602 4581 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 04:47:21.169479 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.168611 4581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 04:47:21.169479 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.169085 4581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 04:47:21.177930 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.177915 4581 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 04:47:21.177930 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.177929 4581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 04:47:21.178057 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.178047 4581 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 04 04:47:21.178250 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.178239 4581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 04:47:21.178278 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.178269 4581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 04:47:21.178380 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.178371 4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 04:47:21.178410 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.178390 4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 04:47:21.178410 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.178400 4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 04:47:21.178410 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.178408 4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 04:47:21.178497 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.178416 4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 04:47:21.178497 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.178424 4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 04:47:21.178497 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.178429 4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 04:47:21.178497 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.178434 4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 04:47:21.178497 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.178441 4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 04:47:21.178497 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.178447 4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 04:47:21.178497 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.178459 4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 04:47:21.178497 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.178468 4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 04:47:21.178497 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.178495 4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 04:47:21.178497 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.178500 4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 04:47:21.181776 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.181760 4581 server.go:1263] "Started kubelet" Nov 04 04:47:21.181890 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.181822 4581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 04:47:21.182091 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.181858 4581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 04:47:21.182143 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.182116 4581 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 04:47:21.182382 ip-10-0-141-132 systemd[1]: Started Kubernetes Kubelet. Nov 04 04:47:21.183643 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.183173 4581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 04:47:21.185138 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.185092 4581 server.go:317] "Adding debug handlers to kubelet server" Nov 04 04:47:21.186181 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.185743 4581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 04:47:21.186181 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.186092 4581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.186316 4581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 04:41:43 +0000 UTC" deadline="2028-05-08 17:55:45.776970887 +0000 UTC" Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.186365 4581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13237h8m24.590617429s" Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.186576 4581 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.186587 4581 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.186703 4581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:47:21.187744 4581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-132.us-west-1.compute.internal\" not found" Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.187937 4581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.187953 4581 factory.go:55] Registering systemd factory Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.187963 4581 factory.go:223] Registration of the systemd container factory successfully Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.188368 4581 factory.go:153] Registering CRI-O factory Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.188381 4581 factory.go:223] Registration of the crio container factory successfully Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.188401 4581 factory.go:103] Registering Raw factory Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.188419 4581 manager.go:1199] Started watching for new ooms in manager Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.188835 4581 manager.go:319] Starting recovery of all containers Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189204 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acddcc2f-2b31-478b-a0de-552929cd7bd4" volumeName="kubernetes.io/configmap/acddcc2f-2b31-478b-a0de-552929cd7bd4-serviceca" seLinuxMountContext="" Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189234 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3e3cff9-0366-457e-a337-32722ca38e00" volumeName="kubernetes.io/configmap/c3e3cff9-0366-457e-a337-32722ca38e00-config-volume" seLinuxMountContext="" Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189265 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e407d116-de8e-4a31-bcee-88546ae61f57" volumeName="kubernetes.io/empty-dir/e407d116-de8e-4a31-bcee-88546ae61f57-data-volume" seLinuxMountContext="" Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189274 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed8f18be-cea2-4c8f-a143-25ddd48b1a74" volumeName="kubernetes.io/projected/ed8f18be-cea2-4c8f-a143-25ddd48b1a74-kube-api-access-w87hs" seLinuxMountContext="" Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189289 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26237d52-9119-41f2-919a-5e5153ea223a" volumeName="kubernetes.io/projected/26237d52-9119-41f2-919a-5e5153ea223a-kube-api-access-nd2j4" seLinuxMountContext="" Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189304 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63e7d159-5f5f-4802-997b-5bfc47f1dad2" volumeName="kubernetes.io/empty-dir/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-tuned" seLinuxMountContext="" Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189320 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acddcc2f-2b31-478b-a0de-552929cd7bd4" volumeName="kubernetes.io/projected/acddcc2f-2b31-478b-a0de-552929cd7bd4-kube-api-access-rjj9c" seLinuxMountContext="" Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189339 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e407d116-de8e-4a31-bcee-88546ae61f57" volumeName="kubernetes.io/configmap/e407d116-de8e-4a31-bcee-88546ae61f57-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189354 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="030a8f10-c4b1-45b2-abb8-88ed48e53a80" volumeName="kubernetes.io/secret/030a8f10-c4b1-45b2-abb8-88ed48e53a80-agent-certs" seLinuxMountContext="" Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189362 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12f1a157-ce5b-4040-98de-ed53507268db" volumeName="kubernetes.io/secret/12f1a157-ce5b-4040-98de-ed53507268db-metrics-certs" seLinuxMountContext="" Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189371 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24bc4691-e47f-4f98-ad47-f14654d95d28" volumeName="kubernetes.io/secret/24bc4691-e47f-4f98-ad47-f14654d95d28-original-pull-secret" seLinuxMountContext="" Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189382 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27a76981-142e-4959-aac0-daaffc4f9b95" volumeName="kubernetes.io/projected/27a76981-142e-4959-aac0-daaffc4f9b95-kube-api-access-xxwbx" seLinuxMountContext="" Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189414 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64661f31-04d2-41e7-aab7-28000a68fc64" volumeName="kubernetes.io/projected/64661f31-04d2-41e7-aab7-28000a68fc64-kube-api-access-q8q7s" seLinuxMountContext="" Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189427 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12f1a157-ce5b-4040-98de-ed53507268db" volumeName="kubernetes.io/projected/12f1a157-ce5b-4040-98de-ed53507268db-kube-api-access-98rdv" seLinuxMountContext="" Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189438 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64661f31-04d2-41e7-aab7-28000a68fc64" volumeName="kubernetes.io/configmap/64661f31-04d2-41e7-aab7-28000a68fc64-ovnkube-script-lib" seLinuxMountContext="" Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189450 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64661f31-04d2-41e7-aab7-28000a68fc64" volumeName="kubernetes.io/secret/64661f31-04d2-41e7-aab7-28000a68fc64-ovn-node-metrics-cert" seLinuxMountContext="" Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189458 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0e2510f-0579-4594-bd3b-a3dadf3fb81a" volumeName="kubernetes.io/projected/d0e2510f-0579-4594-bd3b-a3dadf3fb81a-kube-api-access-ct442" seLinuxMountContext="" Nov 04 04:47:21.189443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189467 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e407d116-de8e-4a31-bcee-88546ae61f57" volumeName="kubernetes.io/secret/e407d116-de8e-4a31-bcee-88546ae61f57-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 04 04:47:21.190901 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189477 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed8f18be-cea2-4c8f-a143-25ddd48b1a74" volumeName="kubernetes.io/empty-dir/ed8f18be-cea2-4c8f-a143-25ddd48b1a74-tmp-dir" seLinuxMountContext="" Nov 04 04:47:21.190901 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189486 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27a76981-142e-4959-aac0-daaffc4f9b95" volumeName="kubernetes.io/configmap/27a76981-142e-4959-aac0-daaffc4f9b95-cni-sysctl-allowlist" seLinuxMountContext="" Nov 04 04:47:21.190901 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189497 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="565b853c-b917-41a5-b739-368db09dac1f" volumeName="kubernetes.io/projected/565b853c-b917-41a5-b739-368db09dac1f-kube-api-access-m9g5g" seLinuxMountContext="" Nov 04 04:47:21.190901 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189509 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63e7d159-5f5f-4802-997b-5bfc47f1dad2" volumeName="kubernetes.io/empty-dir/63e7d159-5f5f-4802-997b-5bfc47f1dad2-tmp" seLinuxMountContext="" Nov 04 04:47:21.190901 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189517 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b" volumeName="kubernetes.io/configmap/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-cni-binary-copy" seLinuxMountContext="" Nov 04 04:47:21.190901 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189527 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3e3cff9-0366-457e-a337-32722ca38e00" volumeName="kubernetes.io/empty-dir/c3e3cff9-0366-457e-a337-32722ca38e00-tmp-dir" seLinuxMountContext="" Nov 04 04:47:21.190901 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189534 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3e3cff9-0366-457e-a337-32722ca38e00" volumeName="kubernetes.io/secret/c3e3cff9-0366-457e-a337-32722ca38e00-metrics-tls" seLinuxMountContext="" Nov 04 04:47:21.190901 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189546 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="030a8f10-c4b1-45b2-abb8-88ed48e53a80" volumeName="kubernetes.io/configmap/030a8f10-c4b1-45b2-abb8-88ed48e53a80-konnectivity-ca" seLinuxMountContext="" Nov 04 04:47:21.190901 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189555 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27a76981-142e-4959-aac0-daaffc4f9b95" volumeName="kubernetes.io/configmap/27a76981-142e-4959-aac0-daaffc4f9b95-cni-binary-copy" seLinuxMountContext="" Nov 04 04:47:21.190901 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189568 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64661f31-04d2-41e7-aab7-28000a68fc64" volumeName="kubernetes.io/configmap/64661f31-04d2-41e7-aab7-28000a68fc64-env-overrides" seLinuxMountContext="" Nov 04 04:47:21.190901 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189597 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b" volumeName="kubernetes.io/configmap/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-multus-daemon-config" seLinuxMountContext="" Nov 04 04:47:21.190901 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189608 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3e3cff9-0366-457e-a337-32722ca38e00" volumeName="kubernetes.io/projected/c3e3cff9-0366-457e-a337-32722ca38e00-kube-api-access-p5s2h" seLinuxMountContext="" Nov 04 04:47:21.190901 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189615 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27a76981-142e-4959-aac0-daaffc4f9b95" volumeName="kubernetes.io/configmap/27a76981-142e-4959-aac0-daaffc4f9b95-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 04 04:47:21.190901 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189627 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bb73603-5b6f-4a77-a39f-033b1f80059f" volumeName="kubernetes.io/secret/3bb73603-5b6f-4a77-a39f-033b1f80059f-cert" seLinuxMountContext="" Nov 04 04:47:21.190901 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189635 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63e7d159-5f5f-4802-997b-5bfc47f1dad2" volumeName="kubernetes.io/projected/63e7d159-5f5f-4802-997b-5bfc47f1dad2-kube-api-access-2jj28" seLinuxMountContext="" Nov 04 04:47:21.190901 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189647 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e407d116-de8e-4a31-bcee-88546ae61f57" volumeName="kubernetes.io/projected/e407d116-de8e-4a31-bcee-88546ae61f57-kube-api-access-s9rtr" seLinuxMountContext="" Nov 04 04:47:21.190901 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189660 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bb73603-5b6f-4a77-a39f-033b1f80059f" volumeName="kubernetes.io/projected/3bb73603-5b6f-4a77-a39f-033b1f80059f-kube-api-access-sjpz9" seLinuxMountContext="" Nov 04 04:47:21.190901 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189668 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="565b853c-b917-41a5-b739-368db09dac1f" volumeName="kubernetes.io/configmap/565b853c-b917-41a5-b739-368db09dac1f-iptables-alerter-script" seLinuxMountContext="" Nov 04 04:47:21.190901 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189678 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64661f31-04d2-41e7-aab7-28000a68fc64" volumeName="kubernetes.io/configmap/64661f31-04d2-41e7-aab7-28000a68fc64-ovnkube-config" seLinuxMountContext="" Nov 04 04:47:21.190901 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189686 4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b" volumeName="kubernetes.io/projected/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-kube-api-access-jjmwq" seLinuxMountContext="" Nov 04 04:47:21.190901 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189694 4581 reconstruct.go:97] "Volume reconstruction finished" Nov 04 04:47:21.190901 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.189699 4581 reconciler.go:29] "Reconciler: start to sync state" Nov 04 04:47:21.221579 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.221549 4581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 04:47:21.223079 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.223061 4581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 04:47:21.223079 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.223083 4581 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 04:47:21.223248 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.223102 4581 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 04:47:21.223248 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:47:21.223145 4581 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 04 04:47:21.240430 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.240403 4581 generic.go:356] "Generic (PLEG): container finished" podID="27a76981-142e-4959-aac0-daaffc4f9b95" containerID="7c827f6e5918fc31fd579c6450d512f4e164a8c799c2a32e0e8b34fcfca65aac" exitCode=0 Nov 04 04:47:21.240430 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.240425 4581 generic.go:356] "Generic (PLEG): container finished" podID="27a76981-142e-4959-aac0-daaffc4f9b95" containerID="372489729f01bb7e98ecc41f4f0a55abf34228f5aa68762a791c3993e22781db" exitCode=0 Nov 04 04:47:21.240430 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.240431 4581 generic.go:356] "Generic (PLEG): container finished" podID="27a76981-142e-4959-aac0-daaffc4f9b95" containerID="3fb4f84439b6598bb978e92cb5ed2e1ca18e0c86345a8b14fdce3277f4cdb70d" exitCode=0 Nov 04 04:47:21.240430 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.240437 4581 generic.go:356] "Generic (PLEG): container finished" podID="27a76981-142e-4959-aac0-daaffc4f9b95" containerID="cbd740d0a140fda47863875b59d967f86cb183ac33093affc5ead75772815e6c" exitCode=0 Nov 04 04:47:21.240695 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.240442 4581 generic.go:356] "Generic (PLEG): container finished" podID="27a76981-142e-4959-aac0-daaffc4f9b95" containerID="cf6e9c76338cd6d6d8b636233f0ff357b51654e656337935d91f8cd39c47df23" exitCode=0 Nov 04 04:47:21.240695 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.240447 4581 generic.go:356] "Generic (PLEG): container finished" podID="27a76981-142e-4959-aac0-daaffc4f9b95" containerID="68868f5b637266aafa28623fb8a59e4f4494e7d2f457c5e47eb3f3c56cd8ebf1" exitCode=0 Nov 04 04:47:21.260825 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.260793 4581 generic.go:356] "Generic (PLEG): container finished" podID="81b6936b6f0cd5f6748dab38c3bd34d9" containerID="e67fa1f4623c936d0c7325002ca2be90cb66c7270745b0110709670f655a1688" exitCode=0 Nov 04 04:47:21.264198 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.264180 4581 manager.go:324] Recovery completed Nov 04 04:47:21.278859 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.278837 4581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:47:21.280195 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.280177 4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:47:21.280271 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.280208 4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:47:21.280271 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.280219 4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:47:21.282427 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.282412 4581 cpu_manager.go:226] "Starting" policy="none" Nov 04 04:47:21.282427 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.282424 4581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 04:47:21.282427 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.282426 4581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:47:21.282565 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.282441 4581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 04:47:21.282594 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.282560 4581 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 04 04:47:21.282594 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.282573 4581 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 04 04:47:21.282648 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.282596 4581 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 04 04:47:21.282648 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.282601 4581 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 04 04:47:21.282648 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.282606 4581 policy_none.go:50] "Start" Nov 04 04:47:21.282648 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.282615 4581 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 04:47:21.282648 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.282624 4581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 04:47:21.282777 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.282748 4581 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 04 04:47:21.282777 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.282755 4581 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 04 04:47:21.282777 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.282761 4581 policy_none.go:44] "Start" Nov 04 04:47:21.283485 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.283471 4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:47:21.283529 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.283503 4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:47:21.283529 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.283518 4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:47:21.286936 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.286914 4581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 04:47:21.287821 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:47:21.287807 4581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-132.us-west-1.compute.internal\" not found" Nov 04 04:47:21.289660 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.289650 4581 manager.go:341] "Starting Device Plugin manager" Nov 04 04:47:21.289713 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:47:21.289679 4581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 04:47:21.289713 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.289687 4581 server.go:87] "Starting device plugin registration server" Nov 04 04:47:21.290028 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.290010 4581 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 04:47:21.290154 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.290027 4581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 04:47:21.290154 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.290101 4581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 04:47:21.290265 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.290184 4581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 04:47:21.290265 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.290221 4581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 04:47:21.290265 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.290228 4581 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 04:47:21.290398 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.290280 4581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T04:47:21.290212869Z","UUID":"56789d0e-305d-43ae-8a57-2f1ee8566427","Handler":null,"Name":"","Endpoint":""} Nov 04 04:47:21.291409 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:47:21.291378 4581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 04:47:21.291486 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:47:21.291428 4581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-132.us-west-1.compute.internal\" not found" Nov 04 04:47:21.291677 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.291660 4581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 04:47:21.291762 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.291686 4581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 04:47:21.323753 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.323700 4581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal"] Nov 04 04:47:21.323846 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.323828 4581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:47:21.324864 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.324849 4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:47:21.324928 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.324876 4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:47:21.324928 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.324886 4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:47:21.325121 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.325112 4581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:47:21.325263 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.325252 4581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:47:21.326010 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.325975 4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:47:21.326089 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.326021 4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:47:21.326089 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.326038 4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:47:21.326089 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.326056 4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:47:21.326089 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.326076 4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:47:21.326089 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.326089 4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:47:21.326267 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:47:21.326252 4581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-132.us-west-1.compute.internal\" not found" node="ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:47:21.326340 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.326315 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal" event={"ID":"e7859ee47f36ca521b3b7f587511c5ea","Type":"ContainerStarted","Data":"2c1cab6bde3566f5877661e42a1dce8cbcbb185d0c7d2edc257c5a214bb2f1c2"} Nov 04 04:47:21.326394 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.326349 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal" event={"ID":"e7859ee47f36ca521b3b7f587511c5ea","Type":"ContainerStarted","Data":"5c1e06c23ceafefa52bd98082bbab3815560178a5df8c33c4d1d034e70ea9f5e"} Nov 04 04:47:21.326435 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.326425 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" event={"ID":"81b6936b6f0cd5f6748dab38c3bd34d9","Type":"ContainerStarted","Data":"7ca2e6ec78c5c9ab0e010fe63d0d4a411fd28cbf1cf9b9584e51bf922aff8ff7"} Nov 04 04:47:21.326435 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.326431 4581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:47:21.326487 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.326440 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" event={"ID":"81b6936b6f0cd5f6748dab38c3bd34d9","Type":"ContainerDied","Data":"e67fa1f4623c936d0c7325002ca2be90cb66c7270745b0110709670f655a1688"} Nov 04 04:47:21.326487 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.326456 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" event={"ID":"81b6936b6f0cd5f6748dab38c3bd34d9","Type":"ContainerStarted","Data":"f3e092af629e4a957bf5377e9721b5f7a1302fdbdb8feba985b3f5fe090cca6d"} Nov 04 04:47:21.327875 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.327857 4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:47:21.327943 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.327885 4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:47:21.327943 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.327895 4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:47:21.328043 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:47:21.328032 4581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-132.us-west-1.compute.internal\" not found" node="ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:47:21.378440 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.378416 4581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:47:21.379190 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.379176 4581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:47:21.380332 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.380305 4581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:47:21.380332 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.380308 4581 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 04 04:47:21.390678 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.390658 4581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:47:21.390728 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.390674 4581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:47:21.397802 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.397781 4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:47:21.397859 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.397809 4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:47:21.397859 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.397818 4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:47:21.397859 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.397850 4581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:47:21.431082 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.431026 4581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:47:21.479281 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.479256 4581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 04:47:21.479385 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:47:21.479310 4581 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:47:21.479385 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.479326 4581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:47:21.480424 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.480406 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/81b6936b6f0cd5f6748dab38c3bd34d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal\" (UID: \"81b6936b6f0cd5f6748dab38c3bd34d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:47:21.480463 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.480433 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81b6936b6f0cd5f6748dab38c3bd34d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal\" (UID: \"81b6936b6f0cd5f6748dab38c3bd34d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:47:21.480463 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.480451 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e7859ee47f36ca521b3b7f587511c5ea-config\") pod \"kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal\" (UID: \"e7859ee47f36ca521b3b7f587511c5ea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:47:21.531165 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.531142 4581 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:47:21.531237 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.531205 4581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:47:21.531736 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.531722 4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeNotReady" Nov 04 04:47:21.531828 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.531758 4581 setters.go:546] "Node became not ready" node="ip-10-0-141-132.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-04T04:47:21Z","lastTransitionTime":"2026-11-04T04:47:21Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 04 04:47:21.538443 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.538419 4581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 04:47:21.538549 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:47:21.538536 4581 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:47:21.580601 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.580577 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/81b6936b6f0cd5f6748dab38c3bd34d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal\" (UID: \"81b6936b6f0cd5f6748dab38c3bd34d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:47:21.580690 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.580610 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81b6936b6f0cd5f6748dab38c3bd34d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal\" (UID: \"81b6936b6f0cd5f6748dab38c3bd34d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:47:21.580690 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.580640 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e7859ee47f36ca521b3b7f587511c5ea-config\") pod \"kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal\" (UID: \"e7859ee47f36ca521b3b7f587511c5ea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:47:21.580690 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.580681 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/81b6936b6f0cd5f6748dab38c3bd34d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal\" (UID: \"81b6936b6f0cd5f6748dab38c3bd34d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:47:21.580925 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.580690 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e7859ee47f36ca521b3b7f587511c5ea-config\") pod \"kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal\" (UID: \"e7859ee47f36ca521b3b7f587511c5ea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:47:21.580925 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.580689 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81b6936b6f0cd5f6748dab38c3bd34d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal\" (UID: \"81b6936b6f0cd5f6748dab38c3bd34d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:47:21.593352 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:21.593326 4581 nodeinfomanager.go:471] existing CSINode "ip-10-0-141-132.us-west-1.compute.internal" is owned by different node (oldNodeID="f462d78e-cc9c-4462-8b89-74b749865e3a", newNodeID=""), cleaning up... Nov 04 04:47:22.168512 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.168484 4581 apiserver.go:52] "Watching apiserver" Nov 04 04:47:22.264527 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.264502 4581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:47:22.275884 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.275859 4581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 04:47:22.276314 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.276296 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-724sb","openshift-ingress-canary/ingress-canary-t9tvv","openshift-multus/multus-cxp8f","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw","kube-system/konnectivity-agent-7csxg","kube-system/kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal","openshift-image-registry/node-ca-lfjtg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal","openshift-ovn-kubernetes/ovnkube-node-l58zg","openshift-cluster-node-tuning-operator/tuned-hlw45","openshift-network-operator/iptables-alerter-znffl","openshift-dns/dns-default-q7xkm","kube-system/global-pull-secret-syncer-m5khv","openshift-dns/node-resolver-wwxmc","openshift-network-diagnostics/network-check-target-6xc6w","openshift-multus/network-metrics-daemon-pbjtk","openshift-multus/multus-additional-cni-plugins-7rkbf"] Nov 04 04:47:22.287848 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.287829 4581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 04:47:22.323928 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.323901 4581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 04:47:22.324066 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:47:22.323955 4581 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.us-west-1.compute.internal" Nov 04 04:47:22.330263 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.330241 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 04:47:22.332421 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.332404 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 04:47:22.332421 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.332406 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mkkxt\"" Nov 04 04:47:22.332421 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.332422 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tqbjc\"" Nov 04 04:47:22.332577 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.332406 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vmp7s\"" Nov 04 04:47:22.332577 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.332469 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g79lj\"" Nov 04 04:47:22.333556 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.333541 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 04:47:22.333702 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.333689 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jqdfn\"" Nov 04 04:47:22.333762 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.333706 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 04:47:22.333762 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.333734 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ssxrh\"" Nov 04 04:47:22.333762 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.333733 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zsr9v\"" Nov 04 04:47:22.333762 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.333732 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 04:47:22.333909 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.333774 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mrmbg\"" Nov 04 04:47:22.367881 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.367858 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-w4xfx\"" Nov 04 04:47:22.369076 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.369056 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 04:47:22.369174 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.369058 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6sxpx\"" Nov 04 04:47:22.369174 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.369130 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 04:47:22.369272 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.369183 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-shz8v\"" Nov 04 04:47:22.369272 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.369245 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jvrn2\"" Nov 04 04:47:22.369377 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.369363 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 04:47:22.369419 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.369370 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-n7w7t\"" Nov 04 04:47:22.369419 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.369393 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 04:47:22.369419 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.369391 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kgq44\"" Nov 04 04:47:22.369513 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.369421 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-q5r74\"" Nov 04 04:47:22.384965 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.384945 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-cni-bin\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.385030 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.384972 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64661f31-04d2-41e7-aab7-28000a68fc64-ovnkube-config\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.385030 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385004 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-run-k8s-cni-cncf-io\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.385030 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385021 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-etc-kubernetes\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.385125 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385069 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-host\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.385125 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385102 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e3cff9-0366-457e-a337-32722ca38e00-config-volume\") pod \"dns-default-q7xkm\" (UID: \"c3e3cff9-0366-457e-a337-32722ca38e00\") " pod="openshift-dns/dns-default-q7xkm" Nov 04 04:47:22.385187 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385155 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5s2h\" (UniqueName: \"kubernetes.io/projected/c3e3cff9-0366-457e-a337-32722ca38e00-kube-api-access-p5s2h\") pod \"dns-default-q7xkm\" (UID: \"c3e3cff9-0366-457e-a337-32722ca38e00\") " pod="openshift-dns/dns-default-q7xkm" Nov 04 04:47:22.385187 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385181 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-run-ovn\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.385253 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385199 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-run-ovn-kubernetes\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.385253 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385221 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed8f18be-cea2-4c8f-a143-25ddd48b1a74-hosts-file\") pod \"node-resolver-wwxmc\" (UID: \"ed8f18be-cea2-4c8f-a143-25ddd48b1a74\") " pod="openshift-dns/node-resolver-wwxmc" Nov 04 04:47:22.385332 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385265 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-sysctl-d\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.385332 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385304 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-systemd\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.385420 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385342 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-socket-dir\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:47:22.385420 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385365 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-etc-selinux\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:47:22.385420 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385407 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-sys-fs\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:47:22.385563 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385448 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e407d116-de8e-4a31-bcee-88546ae61f57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:22.385563 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385482 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e407d116-de8e-4a31-bcee-88546ae61f57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:22.385563 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385508 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-tuned\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.385563 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385532 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxwbx\" (UniqueName: \"kubernetes.io/projected/27a76981-142e-4959-aac0-daaffc4f9b95-kube-api-access-xxwbx\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:47:22.385563 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385556 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed8f18be-cea2-4c8f-a143-25ddd48b1a74-tmp-dir\") pod \"node-resolver-wwxmc\" (UID: \"ed8f18be-cea2-4c8f-a143-25ddd48b1a74\") " pod="openshift-dns/node-resolver-wwxmc" Nov 04 04:47:22.385743 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385581 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-slash\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.385743 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385594 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-tuned\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.385743 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385607 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjpz9\" (UniqueName: \"kubernetes.io/projected/3bb73603-5b6f-4a77-a39f-033b1f80059f-kube-api-access-sjpz9\") pod \"ingress-canary-t9tvv\" (UID: \"3bb73603-5b6f-4a77-a39f-033b1f80059f\") " pod="openshift-ingress-canary/ingress-canary-t9tvv" Nov 04 04:47:22.385743 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385652 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9rtr\" (UniqueName: \"kubernetes.io/projected/e407d116-de8e-4a31-bcee-88546ae61f57-kube-api-access-s9rtr\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:22.385743 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385664 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed8f18be-cea2-4c8f-a143-25ddd48b1a74-tmp-dir\") pod \"node-resolver-wwxmc\" (UID: \"ed8f18be-cea2-4c8f-a143-25ddd48b1a74\") " pod="openshift-dns/node-resolver-wwxmc" Nov 04 04:47:22.385962 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385768 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-run-netns\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.385962 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385802 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/030a8f10-c4b1-45b2-abb8-88ed48e53a80-agent-certs\") pod \"konnectivity-agent-7csxg\" (UID: \"030a8f10-c4b1-45b2-abb8-88ed48e53a80\") " pod="kube-system/konnectivity-agent-7csxg" Nov 04 04:47:22.385962 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385859 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e407d116-de8e-4a31-bcee-88546ae61f57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:22.385962 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385892 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-modprobe-d\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.385962 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385920 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-sysctl-conf\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.385962 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385941 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63e7d159-5f5f-4802-997b-5bfc47f1dad2-tmp\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.385962 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.385961 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-run-systemd\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.386317 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.386030 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjj9c\" (UniqueName: \"kubernetes.io/projected/acddcc2f-2b31-478b-a0de-552929cd7bd4-kube-api-access-rjj9c\") pod \"node-ca-lfjtg\" (UID: \"acddcc2f-2b31-478b-a0de-552929cd7bd4\") " pod="openshift-image-registry/node-ca-lfjtg" Nov 04 04:47:22.386317 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.386037 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63e7d159-5f5f-4802-997b-5bfc47f1dad2-tmp\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.386317 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.386062 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-var-lib-cni-multus\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.386317 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.386168 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjmwq\" (UniqueName: \"kubernetes.io/projected/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-kube-api-access-jjmwq\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.386317 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.386209 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-var-lib-kubelet\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.386317 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.386208 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/030a8f10-c4b1-45b2-abb8-88ed48e53a80-agent-certs\") pod \"konnectivity-agent-7csxg\" (UID: \"030a8f10-c4b1-45b2-abb8-88ed48e53a80\") " pod="kube-system/konnectivity-agent-7csxg" Nov 04 04:47:22.386317 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.386252 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9g5g\" (UniqueName: \"kubernetes.io/projected/565b853c-b917-41a5-b739-368db09dac1f-kube-api-access-m9g5g\") pod \"iptables-alerter-znffl\" (UID: \"565b853c-b917-41a5-b739-368db09dac1f\") " pod="openshift-network-operator/iptables-alerter-znffl" Nov 04 04:47:22.386317 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.386270 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-device-dir\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:47:22.386317 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.386304 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8q7s\" (UniqueName: \"kubernetes.io/projected/64661f31-04d2-41e7-aab7-28000a68fc64-kube-api-access-q8q7s\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.386658 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.386402 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-sys\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.386658 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.386425 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/565b853c-b917-41a5-b739-368db09dac1f-host-slash\") pod \"iptables-alerter-znffl\" (UID: \"565b853c-b917-41a5-b739-368db09dac1f\") " pod="openshift-network-operator/iptables-alerter-znffl" Nov 04 04:47:22.386658 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.386467 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27a76981-142e-4959-aac0-daaffc4f9b95-system-cni-dir\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:47:22.386658 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.386481 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27a76981-142e-4959-aac0-daaffc4f9b95-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:47:22.386658 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.386498 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3e3cff9-0366-457e-a337-32722ca38e00-metrics-tls\") pod \"dns-default-q7xkm\" (UID: \"c3e3cff9-0366-457e-a337-32722ca38e00\") " pod="openshift-dns/dns-default-q7xkm" Nov 04 04:47:22.386658 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.386521 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-multus-daemon-config\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.386658 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.386543 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-lib-modules\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.386658 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.386607 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-registration-dir\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:47:22.386658 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.386648 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-run-openvswitch\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.386894 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.386680 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-log-socket\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.386894 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.386717 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24bc4691-e47f-4f98-ad47-f14654d95d28-kubelet-config\") pod \"global-pull-secret-syncer-m5khv\" (UID: \"24bc4691-e47f-4f98-ad47-f14654d95d28\") " pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:47:22.386894 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.386759 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24bc4691-e47f-4f98-ad47-f14654d95d28-dbus\") pod \"global-pull-secret-syncer-m5khv\" (UID: \"24bc4691-e47f-4f98-ad47-f14654d95d28\") " pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:47:22.386894 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.386780 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3e3cff9-0366-457e-a337-32722ca38e00-metrics-tls\") pod \"dns-default-q7xkm\" (UID: \"c3e3cff9-0366-457e-a337-32722ca38e00\") " pod="openshift-dns/dns-default-q7xkm" Nov 04 04:47:22.386894 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.386796 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98rdv\" (UniqueName: \"kubernetes.io/projected/12f1a157-ce5b-4040-98de-ed53507268db-kube-api-access-98rdv\") pod \"network-metrics-daemon-pbjtk\" (UID: \"12f1a157-ce5b-4040-98de-ed53507268db\") " pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:47:22.386894 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.386827 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-var-lib-cni-bin\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.386894 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.386848 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-multus-conf-dir\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.387119 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387020 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/030a8f10-c4b1-45b2-abb8-88ed48e53a80-konnectivity-ca\") pod \"konnectivity-agent-7csxg\" (UID: \"030a8f10-c4b1-45b2-abb8-88ed48e53a80\") " pod="kube-system/konnectivity-agent-7csxg" Nov 04 04:47:22.387119 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387052 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jj28\" (UniqueName: \"kubernetes.io/projected/63e7d159-5f5f-4802-997b-5bfc47f1dad2-kube-api-access-2jj28\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.387172 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387121 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24bc4691-e47f-4f98-ad47-f14654d95d28-original-pull-secret\") pod \"global-pull-secret-syncer-m5khv\" (UID: \"24bc4691-e47f-4f98-ad47-f14654d95d28\") " pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:47:22.387172 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387159 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64661f31-04d2-41e7-aab7-28000a68fc64-env-overrides\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.387263 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387174 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acddcc2f-2b31-478b-a0de-552929cd7bd4-serviceca\") pod \"node-ca-lfjtg\" (UID: \"acddcc2f-2b31-478b-a0de-552929cd7bd4\") " pod="openshift-image-registry/node-ca-lfjtg" Nov 04 04:47:22.387263 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387188 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3bb73603-5b6f-4a77-a39f-033b1f80059f-cert\") pod \"ingress-canary-t9tvv\" (UID: \"3bb73603-5b6f-4a77-a39f-033b1f80059f\") " pod="openshift-ingress-canary/ingress-canary-t9tvv" Nov 04 04:47:22.387263 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387206 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e407d116-de8e-4a31-bcee-88546ae61f57-crio-socket\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:22.387263 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387237 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-hostroot\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.387391 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387289 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/27a76981-142e-4959-aac0-daaffc4f9b95-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:47:22.387391 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387349 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64661f31-04d2-41e7-aab7-28000a68fc64-env-overrides\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.387506 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387354 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-systemd-units\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.387506 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387407 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24bc4691-e47f-4f98-ad47-f14654d95d28-original-pull-secret\") pod \"global-pull-secret-syncer-m5khv\" (UID: \"24bc4691-e47f-4f98-ad47-f14654d95d28\") " pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:47:22.387506 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387419 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-run-netns\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.387506 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387409 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3bb73603-5b6f-4a77-a39f-033b1f80059f-cert\") pod \"ingress-canary-t9tvv\" (UID: \"3bb73603-5b6f-4a77-a39f-033b1f80059f\") " pod="openshift-ingress-canary/ingress-canary-t9tvv" Nov 04 04:47:22.387506 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387447 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-var-lib-openvswitch\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.387506 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387461 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-etc-openvswitch\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.387506 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387497 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct442\" (UniqueName: \"kubernetes.io/projected/d0e2510f-0579-4594-bd3b-a3dadf3fb81a-kube-api-access-ct442\") pod \"network-check-target-6xc6w\" (UID: \"d0e2510f-0579-4594-bd3b-a3dadf3fb81a\") " pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:47:22.387690 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387520 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e407d116-de8e-4a31-bcee-88546ae61f57-data-volume\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:22.387690 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387564 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-cni-binary-copy\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.387690 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387596 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e407d116-de8e-4a31-bcee-88546ae61f57-data-volume\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:22.387690 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387613 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd2j4\" (UniqueName: \"kubernetes.io/projected/26237d52-9119-41f2-919a-5e5153ea223a-kube-api-access-nd2j4\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:47:22.387690 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387631 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-cnibin\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.387690 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387646 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-multus-socket-dir-parent\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.387690 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387669 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-run-multus-certs\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.387690 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387685 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:47:22.388018 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387791 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27a76981-142e-4959-aac0-daaffc4f9b95-cni-binary-copy\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:47:22.388018 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387844 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-kubelet\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.388018 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387874 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-cni-netd\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.388018 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387935 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.388018 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.387957 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w87hs\" (UniqueName: \"kubernetes.io/projected/ed8f18be-cea2-4c8f-a143-25ddd48b1a74-kube-api-access-w87hs\") pod \"node-resolver-wwxmc\" (UID: \"ed8f18be-cea2-4c8f-a143-25ddd48b1a74\") " pod="openshift-dns/node-resolver-wwxmc" Nov 04 04:47:22.388018 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.388018 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-kubernetes\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.388228 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.388035 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-node-log\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.388228 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.388071 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64661f31-04d2-41e7-aab7-28000a68fc64-ovn-node-metrics-cert\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.388228 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.388097 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f1a157-ce5b-4040-98de-ed53507268db-metrics-certs\") pod \"network-metrics-daemon-pbjtk\" (UID: \"12f1a157-ce5b-4040-98de-ed53507268db\") " pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:47:22.388228 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.388224 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-system-cni-dir\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.388375 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.388308 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-multus-cni-dir\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.388375 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.388350 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64661f31-04d2-41e7-aab7-28000a68fc64-ovn-node-metrics-cert\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.388434 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.388376 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-sysconfig\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.388434 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.388401 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f1a157-ce5b-4040-98de-ed53507268db-metrics-certs\") pod \"network-metrics-daemon-pbjtk\" (UID: \"12f1a157-ce5b-4040-98de-ed53507268db\") " pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:47:22.388434 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.388413 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/27a76981-142e-4959-aac0-daaffc4f9b95-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:47:22.388531 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.388438 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3e3cff9-0366-457e-a337-32722ca38e00-tmp-dir\") pod \"dns-default-q7xkm\" (UID: \"c3e3cff9-0366-457e-a337-32722ca38e00\") " pod="openshift-dns/dns-default-q7xkm" Nov 04 04:47:22.388531 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.388454 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64661f31-04d2-41e7-aab7-28000a68fc64-ovnkube-script-lib\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.388531 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.388468 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acddcc2f-2b31-478b-a0de-552929cd7bd4-host\") pod \"node-ca-lfjtg\" (UID: \"acddcc2f-2b31-478b-a0de-552929cd7bd4\") " pod="openshift-image-registry/node-ca-lfjtg" Nov 04 04:47:22.388531 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.388512 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-var-lib-kubelet\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.388654 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.388537 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3e3cff9-0366-457e-a337-32722ca38e00-tmp-dir\") pod \"dns-default-q7xkm\" (UID: \"c3e3cff9-0366-457e-a337-32722ca38e00\") " pod="openshift-dns/dns-default-q7xkm" Nov 04 04:47:22.388654 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.388538 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-run\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.388654 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.388570 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/565b853c-b917-41a5-b739-368db09dac1f-iptables-alerter-script\") pod \"iptables-alerter-znffl\" (UID: \"565b853c-b917-41a5-b739-368db09dac1f\") " pod="openshift-network-operator/iptables-alerter-znffl" Nov 04 04:47:22.388654 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.388599 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27a76981-142e-4959-aac0-daaffc4f9b95-cnibin\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:47:22.430701 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.430641 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 04:47:22.430701 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.430660 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 04:47:22.430842 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.430766 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 04:47:22.430842 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.430817 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 04:47:22.430842 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.430816 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 04:47:22.430973 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.430905 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 04:47:22.430973 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.430915 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 04:47:22.431113 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.431098 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 04:47:22.431452 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.431422 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 04:47:22.431504 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.431478 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 04:47:22.431504 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.431478 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 04:47:22.431574 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.431515 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 04:47:22.431690 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.431669 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 04:47:22.431781 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.431725 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 04:47:22.431781 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.431755 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 04:47:22.431781 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.431772 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 04:47:22.431934 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.431786 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 04:47:22.431934 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.431791 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 04:47:22.431934 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.431823 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 04:47:22.431934 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.431839 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 04:47:22.431934 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.431840 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 04:47:22.431934 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.431857 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 04:47:22.431934 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.431869 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 04:47:22.431934 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.431866 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 04:47:22.431934 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.431881 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 04:47:22.431934 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.431891 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 04:47:22.431934 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.431893 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 04:47:22.432258 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.431952 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 04:47:22.432258 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.431951 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 04:47:22.432258 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.432044 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 04:47:22.432258 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.432053 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 04:47:22.435241 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.435225 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64661f31-04d2-41e7-aab7-28000a68fc64-ovnkube-config\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.435299 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.435284 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e3cff9-0366-457e-a337-32722ca38e00-config-volume\") pod \"dns-default-q7xkm\" (UID: \"c3e3cff9-0366-457e-a337-32722ca38e00\") " pod="openshift-dns/dns-default-q7xkm" Nov 04 04:47:22.436439 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.436425 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e407d116-de8e-4a31-bcee-88546ae61f57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:22.437607 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.437592 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/030a8f10-c4b1-45b2-abb8-88ed48e53a80-konnectivity-ca\") pod \"konnectivity-agent-7csxg\" (UID: \"030a8f10-c4b1-45b2-abb8-88ed48e53a80\") " pod="kube-system/konnectivity-agent-7csxg" Nov 04 04:47:22.437795 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.437778 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/27a76981-142e-4959-aac0-daaffc4f9b95-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:47:22.437855 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.437820 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-multus-daemon-config\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.437900 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.437879 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-cni-binary-copy\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.437999 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.437972 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27a76981-142e-4959-aac0-daaffc4f9b95-cni-binary-copy\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:47:22.438077 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.438063 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acddcc2f-2b31-478b-a0de-552929cd7bd4-serviceca\") pod \"node-ca-lfjtg\" (UID: \"acddcc2f-2b31-478b-a0de-552929cd7bd4\") " pod="openshift-image-registry/node-ca-lfjtg" Nov 04 04:47:22.439018 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.439002 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/27a76981-142e-4959-aac0-daaffc4f9b95-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:47:22.439062 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.439019 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64661f31-04d2-41e7-aab7-28000a68fc64-ovnkube-script-lib\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.439092 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.439068 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/565b853c-b917-41a5-b739-368db09dac1f-iptables-alerter-script\") pod \"iptables-alerter-znffl\" (UID: \"565b853c-b917-41a5-b739-368db09dac1f\") " pod="openshift-network-operator/iptables-alerter-znffl" Nov 04 04:47:22.440822 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.440803 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxwbx\" (UniqueName: \"kubernetes.io/projected/27a76981-142e-4959-aac0-daaffc4f9b95-kube-api-access-xxwbx\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:47:22.440902 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.440859 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjpz9\" (UniqueName: \"kubernetes.io/projected/3bb73603-5b6f-4a77-a39f-033b1f80059f-kube-api-access-sjpz9\") pod \"ingress-canary-t9tvv\" (UID: \"3bb73603-5b6f-4a77-a39f-033b1f80059f\") " pod="openshift-ingress-canary/ingress-canary-t9tvv" Nov 04 04:47:22.440902 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.440897 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5s2h\" (UniqueName: \"kubernetes.io/projected/c3e3cff9-0366-457e-a337-32722ca38e00-kube-api-access-p5s2h\") pod \"dns-default-q7xkm\" (UID: \"c3e3cff9-0366-457e-a337-32722ca38e00\") " pod="openshift-dns/dns-default-q7xkm" Nov 04 04:47:22.442147 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.442128 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9rtr\" (UniqueName: \"kubernetes.io/projected/e407d116-de8e-4a31-bcee-88546ae61f57-kube-api-access-s9rtr\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:22.442291 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.442266 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8q7s\" (UniqueName: \"kubernetes.io/projected/64661f31-04d2-41e7-aab7-28000a68fc64-kube-api-access-q8q7s\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.442608 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.442267 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9g5g\" (UniqueName: \"kubernetes.io/projected/565b853c-b917-41a5-b739-368db09dac1f-kube-api-access-m9g5g\") pod \"iptables-alerter-znffl\" (UID: \"565b853c-b917-41a5-b739-368db09dac1f\") " pod="openshift-network-operator/iptables-alerter-znffl" Nov 04 04:47:22.446392 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.446265 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjj9c\" (UniqueName: \"kubernetes.io/projected/acddcc2f-2b31-478b-a0de-552929cd7bd4-kube-api-access-rjj9c\") pod \"node-ca-lfjtg\" (UID: \"acddcc2f-2b31-478b-a0de-552929cd7bd4\") " pod="openshift-image-registry/node-ca-lfjtg" Nov 04 04:47:22.446392 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.446304 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98rdv\" (UniqueName: \"kubernetes.io/projected/12f1a157-ce5b-4040-98de-ed53507268db-kube-api-access-98rdv\") pod \"network-metrics-daemon-pbjtk\" (UID: \"12f1a157-ce5b-4040-98de-ed53507268db\") " pod="openshift-multus/network-metrics-daemon-pbjtk" Nov 04 04:47:22.446392 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.446313 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjmwq\" (UniqueName: \"kubernetes.io/projected/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-kube-api-access-jjmwq\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.446392 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.446338 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jj28\" (UniqueName: \"kubernetes.io/projected/63e7d159-5f5f-4802-997b-5bfc47f1dad2-kube-api-access-2jj28\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.446392 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.446353 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd2j4\" (UniqueName: \"kubernetes.io/projected/26237d52-9119-41f2-919a-5e5153ea223a-kube-api-access-nd2j4\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:47:22.446662 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.446398 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w87hs\" (UniqueName: \"kubernetes.io/projected/ed8f18be-cea2-4c8f-a143-25ddd48b1a74-kube-api-access-w87hs\") pod \"node-resolver-wwxmc\" (UID: \"ed8f18be-cea2-4c8f-a143-25ddd48b1a74\") " pod="openshift-dns/node-resolver-wwxmc" Nov 04 04:47:22.446662 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.446565 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct442\" (UniqueName: \"kubernetes.io/projected/d0e2510f-0579-4594-bd3b-a3dadf3fb81a-kube-api-access-ct442\") pod \"network-check-target-6xc6w\" (UID: \"d0e2510f-0579-4594-bd3b-a3dadf3fb81a\") " pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:47:22.489627 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489600 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-etc-selinux\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:47:22.489627 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489627 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-sys-fs\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:47:22.489739 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489643 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-slash\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.489739 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489666 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-run-netns\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.489739 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489713 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-run-netns\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.489739 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489713 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-etc-selinux\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:47:22.489882 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489730 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-sys-fs\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:47:22.489882 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489737 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-modprobe-d\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.489882 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489732 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-slash\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.489882 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489769 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-sysctl-conf\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.489882 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489791 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-run-systemd\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.489882 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489806 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-var-lib-cni-multus\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.489882 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489826 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-var-lib-kubelet\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.489882 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489829 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-modprobe-d\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.489882 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489846 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-device-dir\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:47:22.489882 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489875 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-run-systemd\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489880 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-sys\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489897 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-device-dir\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489900 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-var-lib-kubelet\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489914 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-sys\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489917 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-sysctl-conf\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489912 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-var-lib-cni-multus\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489919 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/565b853c-b917-41a5-b739-368db09dac1f-host-slash\") pod \"iptables-alerter-znffl\" (UID: \"565b853c-b917-41a5-b739-368db09dac1f\") " pod="openshift-network-operator/iptables-alerter-znffl" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489944 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/565b853c-b917-41a5-b739-368db09dac1f-host-slash\") pod \"iptables-alerter-znffl\" (UID: \"565b853c-b917-41a5-b739-368db09dac1f\") " pod="openshift-network-operator/iptables-alerter-znffl" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489953 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27a76981-142e-4959-aac0-daaffc4f9b95-system-cni-dir\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.489969 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27a76981-142e-4959-aac0-daaffc4f9b95-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490021 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27a76981-142e-4959-aac0-daaffc4f9b95-system-cni-dir\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490032 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27a76981-142e-4959-aac0-daaffc4f9b95-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490005 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-lib-modules\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490061 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-registration-dir\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490076 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-run-openvswitch\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490090 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-log-socket\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490103 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24bc4691-e47f-4f98-ad47-f14654d95d28-kubelet-config\") pod \"global-pull-secret-syncer-m5khv\" (UID: \"24bc4691-e47f-4f98-ad47-f14654d95d28\") " pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490108 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-lib-modules\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490117 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24bc4691-e47f-4f98-ad47-f14654d95d28-dbus\") pod \"global-pull-secret-syncer-m5khv\" (UID: \"24bc4691-e47f-4f98-ad47-f14654d95d28\") " pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490127 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-registration-dir\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490133 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-var-lib-cni-bin\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490157 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-run-openvswitch\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490167 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-var-lib-cni-bin\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490134 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-log-socket\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490178 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24bc4691-e47f-4f98-ad47-f14654d95d28-kubelet-config\") pod \"global-pull-secret-syncer-m5khv\" (UID: \"24bc4691-e47f-4f98-ad47-f14654d95d28\") " pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490184 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-multus-conf-dir\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490190 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24bc4691-e47f-4f98-ad47-f14654d95d28-dbus\") pod \"global-pull-secret-syncer-m5khv\" (UID: \"24bc4691-e47f-4f98-ad47-f14654d95d28\") " pod="kube-system/global-pull-secret-syncer-m5khv" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490219 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e407d116-de8e-4a31-bcee-88546ae61f57-crio-socket\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490220 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-multus-conf-dir\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490234 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-hostroot\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490250 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-systemd-units\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490262 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-hostroot\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490268 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e407d116-de8e-4a31-bcee-88546ae61f57-crio-socket\") pod \"insights-runtime-extractor-724sb\" (UID: \"e407d116-de8e-4a31-bcee-88546ae61f57\") " pod="openshift-insights/insights-runtime-extractor-724sb" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490280 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-systemd-units\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490287 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-run-netns\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490265 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-run-netns\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.490303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490310 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-var-lib-openvswitch\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490329 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-etc-openvswitch\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490348 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-cnibin\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490362 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-multus-socket-dir-parent\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490378 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-etc-openvswitch\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490379 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-run-multus-certs\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490381 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-var-lib-openvswitch\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490400 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-cnibin\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490402 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490426 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-run-multus-certs\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490428 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-multus-socket-dir-parent\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490429 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-kubelet\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490462 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-cni-netd\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490466 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-kubelet\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490477 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490489 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490509 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-kubernetes\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490523 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-node-log\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490540 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-system-cni-dir\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490556 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490562 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-multus-cni-dir\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490579 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-node-log\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490586 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-cni-netd\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490590 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-sysconfig\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490609 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-multus-cni-dir\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490614 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-system-cni-dir\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490616 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acddcc2f-2b31-478b-a0de-552929cd7bd4-host\") pod \"node-ca-lfjtg\" (UID: \"acddcc2f-2b31-478b-a0de-552929cd7bd4\") " pod="openshift-image-registry/node-ca-lfjtg" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490643 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acddcc2f-2b31-478b-a0de-552929cd7bd4-host\") pod \"node-ca-lfjtg\" (UID: \"acddcc2f-2b31-478b-a0de-552929cd7bd4\") " pod="openshift-image-registry/node-ca-lfjtg" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490630 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-sysconfig\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490654 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-var-lib-kubelet\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490676 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-run\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490682 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-var-lib-kubelet\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490581 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-kubernetes\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490693 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27a76981-142e-4959-aac0-daaffc4f9b95-cnibin\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490708 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-cni-bin\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490714 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-run\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490726 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-run-k8s-cni-cncf-io\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490749 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-etc-kubernetes\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490754 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27a76981-142e-4959-aac0-daaffc4f9b95-cnibin\") pod \"multus-additional-cni-plugins-7rkbf\" (UID: \"27a76981-142e-4959-aac0-daaffc4f9b95\") " pod="openshift-multus/multus-additional-cni-plugins-7rkbf" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490770 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-host\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490786 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-cni-bin\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490793 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-etc-kubernetes\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490794 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-run-ovn\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490793 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b-host-run-k8s-cni-cncf-io\") pod \"multus-cxp8f\" (UID: \"8b9a4dfc-3b04-4a71-ab1c-dd5da4b22c8b\") " pod="openshift-multus/multus-cxp8f" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490823 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-host\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490846 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-run-ovn-kubernetes\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490869 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed8f18be-cea2-4c8f-a143-25ddd48b1a74-hosts-file\") pod \"node-resolver-wwxmc\" (UID: \"ed8f18be-cea2-4c8f-a143-25ddd48b1a74\") " pod="openshift-dns/node-resolver-wwxmc" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490888 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-sysctl-d\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490890 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-run-ovn\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490910 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-systemd\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490913 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64661f31-04d2-41e7-aab7-28000a68fc64-host-run-ovn-kubernetes\") pod \"ovnkube-node-l58zg\" (UID: \"64661f31-04d2-41e7-aab7-28000a68fc64\") " pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490929 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-socket-dir\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490956 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed8f18be-cea2-4c8f-a143-25ddd48b1a74-hosts-file\") pod \"node-resolver-wwxmc\" (UID: \"ed8f18be-cea2-4c8f-a143-25ddd48b1a74\") " pod="openshift-dns/node-resolver-wwxmc" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.490976 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-systemd\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.491011 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/26237d52-9119-41f2-919a-5e5153ea223a-socket-dir\") pod \"aws-ebs-csi-driver-node-jxfjw\" (UID: \"26237d52-9119-41f2-919a-5e5153ea223a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jxfjw" Nov 04 04:47:22.491446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.491027 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63e7d159-5f5f-4802-997b-5bfc47f1dad2-etc-sysctl-d\") pod \"tuned-hlw45\" (UID: \"63e7d159-5f5f-4802-997b-5bfc47f1dad2\") " pod="openshift-cluster-node-tuning-operator/tuned-hlw45" Nov 04 04:47:22.583912 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.583891 4581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 04:47:22.842484 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.842412 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:22.959306 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:22.959275 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:24.273942 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:24.273916 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-724sb" event={"ID":"e407d116-de8e-4a31-bcee-88546ae61f57","Type":"ContainerStarted","Data":"86368199c5f7768d2fd88c0ae7d3c59c435141be9e1c2b14fcc466220107617d"} Nov 04 04:47:24.747022 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:24.746972 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-q7xkm" Nov 04 04:47:24.812586 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:24.812561 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:47:27.285156 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:27.285120 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-724sb" event={"ID":"e407d116-de8e-4a31-bcee-88546ae61f57","Type":"ContainerStarted","Data":"253cbe7f71fbbdfcc28ae19867efdebfa30671b6da22f4e07fb01bfd6ba5d507"} Nov 04 04:47:27.749803 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:27.749776 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q7xkm" Nov 04 04:47:28.527817 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:28.527789 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7csxg" Nov 04 04:47:30.492088 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:30.492056 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:30.664440 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:30.664407 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:30.798191 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:30.798107 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:30.812693 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:30.812670 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:30.918015 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:30.917964 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:31.413355 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:31.413326 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:31.413559 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:31.413535 4581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 04:47:31.921223 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:31.921196 4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-132.us-west-1.compute.internal" event="NodeReady" Nov 04 04:47:33.874280 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:33.874242 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:34.040265 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.040226 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:34.160977 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.160895 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:34.291652 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.291482 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-d9tqz"] Nov 04 04:47:34.293378 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.293356 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-d9tqz" Nov 04 04:47:34.361822 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.361783 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1580245-c2af-4604-870c-390f2bdc22c5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-d9tqz\" (UID: \"f1580245-c2af-4604-870c-390f2bdc22c5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-d9tqz" Nov 04 04:47:34.361980 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.361837 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1580245-c2af-4604-870c-390f2bdc22c5-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-d9tqz\" (UID: \"f1580245-c2af-4604-870c-390f2bdc22c5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-d9tqz" Nov 04 04:47:34.361980 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.361956 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1580245-c2af-4604-870c-390f2bdc22c5-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-d9tqz\" (UID: \"f1580245-c2af-4604-870c-390f2bdc22c5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-d9tqz" Nov 04 04:47:34.362111 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.362000 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv5gk\" (UniqueName: \"kubernetes.io/projected/f1580245-c2af-4604-870c-390f2bdc22c5-kube-api-access-dv5gk\") pod \"prometheus-operator-9cd89b74c-d9tqz\" (UID: \"f1580245-c2af-4604-870c-390f2bdc22c5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-d9tqz" Nov 04 04:47:34.396715 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.396680 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 04 04:47:34.396715 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.396699 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 04 04:47:34.396882 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.396723 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 04 04:47:34.396882 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.396798 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 04 04:47:34.409614 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.409588 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:34.437649 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.437599 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 04 04:47:34.437649 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.437624 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-l9mww\"" Nov 04 04:47:34.447564 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.447544 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-d9tqz"] Nov 04 04:47:34.463032 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.463009 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1580245-c2af-4604-870c-390f2bdc22c5-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-d9tqz\" (UID: \"f1580245-c2af-4604-870c-390f2bdc22c5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-d9tqz" Nov 04 04:47:34.463032 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.463035 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv5gk\" (UniqueName: \"kubernetes.io/projected/f1580245-c2af-4604-870c-390f2bdc22c5-kube-api-access-dv5gk\") pod \"prometheus-operator-9cd89b74c-d9tqz\" (UID: \"f1580245-c2af-4604-870c-390f2bdc22c5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-d9tqz" Nov 04 04:47:34.463196 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.463072 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1580245-c2af-4604-870c-390f2bdc22c5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-d9tqz\" (UID: \"f1580245-c2af-4604-870c-390f2bdc22c5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-d9tqz" Nov 04 04:47:34.463196 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.463092 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1580245-c2af-4604-870c-390f2bdc22c5-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-d9tqz\" (UID: \"f1580245-c2af-4604-870c-390f2bdc22c5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-d9tqz" Nov 04 04:47:34.463401 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.463384 4581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 04:47:34.463698 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.463677 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1580245-c2af-4604-870c-390f2bdc22c5-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-d9tqz\" (UID: \"f1580245-c2af-4604-870c-390f2bdc22c5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-d9tqz" Nov 04 04:47:34.465527 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.465508 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1580245-c2af-4604-870c-390f2bdc22c5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-d9tqz\" (UID: \"f1580245-c2af-4604-870c-390f2bdc22c5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-d9tqz" Nov 04 04:47:34.465612 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.465602 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1580245-c2af-4604-870c-390f2bdc22c5-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-d9tqz\" (UID: \"f1580245-c2af-4604-870c-390f2bdc22c5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-d9tqz" Nov 04 04:47:34.520073 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.520042 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv5gk\" (UniqueName: \"kubernetes.io/projected/f1580245-c2af-4604-870c-390f2bdc22c5-kube-api-access-dv5gk\") pod \"prometheus-operator-9cd89b74c-d9tqz\" (UID: \"f1580245-c2af-4604-870c-390f2bdc22c5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-d9tqz" Nov 04 04:47:34.605811 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.605773 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-d9tqz" Nov 04 04:47:34.972591 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:34.972561 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-d9tqz"] Nov 04 04:47:34.975845 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:34.975802 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1580245_c2af_4604_870c_390f2bdc22c5.slice/crio-f1b3badabee21bdc27216fcb446436ee04a70b2a7593eeed67ead8f649bfbbac WatchSource:0}: Error finding container f1b3badabee21bdc27216fcb446436ee04a70b2a7593eeed67ead8f649bfbbac: Status 404 returned error can't find the container with id f1b3badabee21bdc27216fcb446436ee04a70b2a7593eeed67ead8f649bfbbac Nov 04 04:47:35.311584 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:35.311493 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-d9tqz" event={"ID":"f1580245-c2af-4604-870c-390f2bdc22c5","Type":"ContainerStarted","Data":"f1b3badabee21bdc27216fcb446436ee04a70b2a7593eeed67ead8f649bfbbac"} Nov 04 04:47:37.319378 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:37.319340 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-d9tqz" event={"ID":"f1580245-c2af-4604-870c-390f2bdc22c5","Type":"ContainerStarted","Data":"cd4406e9c6e2e4c02486bb1e7fd0f900b7fbb64fbba5baeca2b7c5bca0129847"} Nov 04 04:47:37.319864 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:37.319404 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-d9tqz" event={"ID":"f1580245-c2af-4604-870c-390f2bdc22c5","Type":"ContainerStarted","Data":"52db2495964ef80b1ef9084f42b9623fe3b04e2710f717ae57597f0cbd4137ed"} Nov 04 04:47:37.437087 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:37.437019 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-d9tqz" podStartSLOduration=1.6895144370000001 podStartE2EDuration="3.437004451s" totalImagesPullingTime="1.747490014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:47:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:37.436723902 +0000 UTC m=+16.327580921" watchObservedRunningTime="2026-11-04 04:47:37.437004451 +0000 UTC m=+16.327861462" Nov 04 04:47:37.546956 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:37.546918 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:37.667586 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:37.667553 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:37.845801 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:37.845771 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:37.967041 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:37.966951 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:39.397686 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:39.397646 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:39.570301 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:39.570269 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:40.111141 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:40.111107 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:40.345599 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:40.345569 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:40.565807 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:40.565726 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:40.715645 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:40.715615 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:40.885875 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:40.885846 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rpwmg"] Nov 04 04:47:40.887869 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:40.887848 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:40.895718 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:40.895691 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:40.908411 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:40.908383 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-metrics-client-ca\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:40.908510 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:40.908424 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-node-exporter-accelerators-collector-config\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:40.908510 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:40.908449 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-node-exporter-wtmp\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:40.908592 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:40.908533 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-sys\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:40.908592 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:40.908559 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-node-exporter-tls\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:40.908592 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:40.908575 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:40.908703 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:40.908599 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-root\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:40.908703 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:40.908689 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-node-exporter-textfile\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:40.908800 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:40.908709 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k6q7\" (UniqueName: \"kubernetes.io/projected/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-kube-api-access-8k6q7\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:40.936318 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:40.936292 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2"] Nov 04 04:47:40.938099 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:40.938086 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2" Nov 04 04:47:40.986230 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:40.986202 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dxmws\"" Nov 04 04:47:40.986230 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:40.986203 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 04 04:47:40.986406 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:40.986204 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 04 04:47:40.986406 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:40.986261 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 04 04:47:40.991450 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:40.991428 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2"] Nov 04 04:47:41.009450 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.009419 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-sys\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:41.009559 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.009461 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a3d46a1d-9630-4f75-97c8-09302e92d701-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hklc2\" (UID: \"a3d46a1d-9630-4f75-97c8-09302e92d701\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2" Nov 04 04:47:41.009559 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.009494 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8dt8\" (UniqueName: \"kubernetes.io/projected/a3d46a1d-9630-4f75-97c8-09302e92d701-kube-api-access-g8dt8\") pod \"kube-state-metrics-5dc8d7d89b-hklc2\" (UID: \"a3d46a1d-9630-4f75-97c8-09302e92d701\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2" Nov 04 04:47:41.009559 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.009516 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-sys\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:41.009721 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.009601 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-node-exporter-textfile\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:41.009721 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.009640 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3d46a1d-9630-4f75-97c8-09302e92d701-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hklc2\" (UID: \"a3d46a1d-9630-4f75-97c8-09302e92d701\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2" Nov 04 04:47:41.009721 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.009673 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3d46a1d-9630-4f75-97c8-09302e92d701-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hklc2\" (UID: \"a3d46a1d-9630-4f75-97c8-09302e92d701\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2" Nov 04 04:47:41.009721 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.009708 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-node-exporter-accelerators-collector-config\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:41.009911 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.009737 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-metrics-client-ca\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:41.009911 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.009838 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-node-exporter-wtmp\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:41.009911 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.009876 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a3d46a1d-9630-4f75-97c8-09302e92d701-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hklc2\" (UID: \"a3d46a1d-9630-4f75-97c8-09302e92d701\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2" Nov 04 04:47:41.010081 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.009911 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a3d46a1d-9630-4f75-97c8-09302e92d701-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hklc2\" (UID: \"a3d46a1d-9630-4f75-97c8-09302e92d701\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2" Nov 04 04:47:41.010081 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.009961 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-node-exporter-tls\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:41.010081 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.010015 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:41.010081 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.010030 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-node-exporter-textfile\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:41.010081 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.010045 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-root\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:41.010081 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.010062 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-node-exporter-wtmp\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:41.010437 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.010097 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-root\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:41.010437 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.010116 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k6q7\" (UniqueName: \"kubernetes.io/projected/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-kube-api-access-8k6q7\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:41.010629 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.010606 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-metrics-client-ca\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:41.010693 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.010641 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-node-exporter-accelerators-collector-config\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:41.012444 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.012426 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-node-exporter-tls\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:41.013114 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.013097 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:41.020703 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.020688 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l58zg" Nov 04 04:47:41.037701 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.037682 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 04 04:47:41.038028 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.038011 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 04 04:47:41.038028 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.038020 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-cz8zl\"" Nov 04 04:47:41.045497 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.045483 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 04 04:47:41.066748 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.066730 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k6q7\" (UniqueName: \"kubernetes.io/projected/d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c-kube-api-access-8k6q7\") pod \"node-exporter-rpwmg\" (UID: \"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c\") " pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:41.086599 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.086560 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:41.111201 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.111178 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a3d46a1d-9630-4f75-97c8-09302e92d701-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hklc2\" (UID: \"a3d46a1d-9630-4f75-97c8-09302e92d701\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2" Nov 04 04:47:41.111272 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.111205 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8dt8\" (UniqueName: \"kubernetes.io/projected/a3d46a1d-9630-4f75-97c8-09302e92d701-kube-api-access-g8dt8\") pod \"kube-state-metrics-5dc8d7d89b-hklc2\" (UID: \"a3d46a1d-9630-4f75-97c8-09302e92d701\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2" Nov 04 04:47:41.111272 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.111236 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3d46a1d-9630-4f75-97c8-09302e92d701-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hklc2\" (UID: \"a3d46a1d-9630-4f75-97c8-09302e92d701\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2" Nov 04 04:47:41.111272 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.111264 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3d46a1d-9630-4f75-97c8-09302e92d701-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hklc2\" (UID: \"a3d46a1d-9630-4f75-97c8-09302e92d701\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2" Nov 04 04:47:41.111415 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.111311 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a3d46a1d-9630-4f75-97c8-09302e92d701-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hklc2\" (UID: \"a3d46a1d-9630-4f75-97c8-09302e92d701\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2" Nov 04 04:47:41.111415 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.111334 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a3d46a1d-9630-4f75-97c8-09302e92d701-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hklc2\" (UID: \"a3d46a1d-9630-4f75-97c8-09302e92d701\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2" Nov 04 04:47:41.111848 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.111816 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a3d46a1d-9630-4f75-97c8-09302e92d701-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hklc2\" (UID: \"a3d46a1d-9630-4f75-97c8-09302e92d701\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2" Nov 04 04:47:41.112016 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.111973 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a3d46a1d-9630-4f75-97c8-09302e92d701-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hklc2\" (UID: \"a3d46a1d-9630-4f75-97c8-09302e92d701\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2" Nov 04 04:47:41.112077 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.112034 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3d46a1d-9630-4f75-97c8-09302e92d701-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hklc2\" (UID: \"a3d46a1d-9630-4f75-97c8-09302e92d701\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2" Nov 04 04:47:41.113737 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.113716 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3d46a1d-9630-4f75-97c8-09302e92d701-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hklc2\" (UID: \"a3d46a1d-9630-4f75-97c8-09302e92d701\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2" Nov 04 04:47:41.113828 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.113812 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a3d46a1d-9630-4f75-97c8-09302e92d701-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hklc2\" (UID: \"a3d46a1d-9630-4f75-97c8-09302e92d701\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2" Nov 04 04:47:41.169361 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.169304 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8dt8\" (UniqueName: \"kubernetes.io/projected/a3d46a1d-9630-4f75-97c8-09302e92d701-kube-api-access-g8dt8\") pod \"kube-state-metrics-5dc8d7d89b-hklc2\" (UID: \"a3d46a1d-9630-4f75-97c8-09302e92d701\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2" Nov 04 04:47:41.201371 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.201343 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rpwmg" Nov 04 04:47:41.212385 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:41.212360 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd016d731_0fb7_4f5a_8f30_3d3e5ee5be6c.slice/crio-059b1c6ca3f16df1c01e0fe8c316d5de7844c611328b378657b9532590b0e6e6 WatchSource:0}: Error finding container 059b1c6ca3f16df1c01e0fe8c316d5de7844c611328b378657b9532590b0e6e6: Status 404 returned error can't find the container with id 059b1c6ca3f16df1c01e0fe8c316d5de7844c611328b378657b9532590b0e6e6 Nov 04 04:47:41.249805 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.249776 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2" Nov 04 04:47:41.290657 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.290615 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:41.331606 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.331573 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rpwmg" event={"ID":"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c","Type":"ContainerStarted","Data":"059b1c6ca3f16df1c01e0fe8c316d5de7844c611328b378657b9532590b0e6e6"} Nov 04 04:47:41.472855 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.472765 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:41.666502 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:41.666474 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2"] Nov 04 04:47:41.669459 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:41.669436 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d46a1d_9630_4f75_97c8_09302e92d701.slice/crio-4cc8d4e60300a5a9c2f0dff6392bcea28f18b8911f7e5315ceddfb352e79740c WatchSource:0}: Error finding container 4cc8d4e60300a5a9c2f0dff6392bcea28f18b8911f7e5315ceddfb352e79740c: Status 404 returned error can't find the container with id 4cc8d4e60300a5a9c2f0dff6392bcea28f18b8911f7e5315ceddfb352e79740c Nov 04 04:47:42.097495 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:42.097461 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:42.281473 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:42.281439 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:42.336677 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:42.336640 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2" event={"ID":"a3d46a1d-9630-4f75-97c8-09302e92d701","Type":"ContainerStarted","Data":"4cc8d4e60300a5a9c2f0dff6392bcea28f18b8911f7e5315ceddfb352e79740c"} Nov 04 04:47:42.960643 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:42.960612 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:43.054203 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.054175 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 04:47:43.058721 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.058025 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.071015 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.070618 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 04:47:43.120787 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.119886 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 04 04:47:43.120787 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.119909 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 04 04:47:43.120787 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.119909 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 04 04:47:43.120787 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.119916 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 04 04:47:43.120787 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.119959 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 04 04:47:43.120787 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.120003 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 04 04:47:43.120787 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.120005 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 04 04:47:43.120787 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.120045 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-chrbj\"" Nov 04 04:47:43.126477 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.126437 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.126617 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.126502 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.126617 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.126538 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89k7r\" (UniqueName: \"kubernetes.io/projected/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-kube-api-access-89k7r\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.126617 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.126600 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.126801 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.126656 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.126801 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.126710 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.126801 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.126777 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-config-volume\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.126934 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.126805 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.126934 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.126829 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-config-out\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.126934 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.126891 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.127108 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.126961 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.127108 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.127066 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-web-config\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.127188 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.127109 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.227839 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.227753 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.227839 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.227795 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89k7r\" (UniqueName: \"kubernetes.io/projected/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-kube-api-access-89k7r\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.227839 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.227826 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.228153 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.227853 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.228153 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.227877 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.228153 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.227909 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-config-volume\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.228153 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.227935 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.228153 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.227961 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-config-out\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.228153 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.228003 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.228153 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.228029 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.228153 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.228072 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-web-config\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.228153 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.228101 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.228153 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.228149 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.228634 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.228292 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.228706 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.228676 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.231564 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.231544 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.232416 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.232392 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-config-out\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.232524 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.232429 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.232524 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.232515 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.232761 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.232741 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-web-config\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.233132 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.233113 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-config-volume\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.233218 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.233190 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.234357 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.234339 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.244168 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.244149 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 04 04:47:43.244359 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.244337 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 04 04:47:43.250000 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.249966 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.250222 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.250200 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.285613 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.285579 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89k7r\" (UniqueName: \"kubernetes.io/projected/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-kube-api-access-89k7r\") pod \"alertmanager-main-0\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.346603 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.346579 4581 generic.go:356] "Generic (PLEG): container finished" podID="d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c" containerID="3d60e1777252664c3fa0a359884408bbd138d67bea565028fce23407f1effd76" exitCode=0 Nov 04 04:47:43.346710 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.346620 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rpwmg" event={"ID":"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c","Type":"ContainerDied","Data":"3d60e1777252664c3fa0a359884408bbd138d67bea565028fce23407f1effd76"} Nov 04 04:47:43.378638 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.378611 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:47:43.469017 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.468570 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7976f76979-72mkk"] Nov 04 04:47:43.471697 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.471674 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.496665 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.496620 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:43.524934 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.524909 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 04 04:47:43.525071 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.524946 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-934gt6gb9f9i5\"" Nov 04 04:47:43.525071 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.524967 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 04 04:47:43.525071 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.525004 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-s8tfq\"" Nov 04 04:47:43.525071 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.525055 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 04 04:47:43.525071 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.525070 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 04 04:47:43.525250 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.525086 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 04 04:47:43.530505 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.530484 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-secret-thanos-querier-tls\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.530601 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.530522 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.530601 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.530548 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-tmp\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.530601 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.530562 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-secret-grpc-tls\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.530765 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.530691 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.530765 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.530739 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.530864 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.530769 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-metrics-client-ca\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.530864 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.530795 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2xpb\" (UniqueName: \"kubernetes.io/projected/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-kube-api-access-l2xpb\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.530864 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.530833 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.536540 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.536448 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7976f76979-72mkk"] Nov 04 04:47:43.631472 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.631444 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-tmp\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.631472 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.631475 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-secret-grpc-tls\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.631731 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.631501 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.631731 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.631522 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.631731 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.631541 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-metrics-client-ca\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.631731 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.631560 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2xpb\" (UniqueName: \"kubernetes.io/projected/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-kube-api-access-l2xpb\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.631731 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.631690 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.631959 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.631753 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-secret-thanos-querier-tls\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.631959 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.631807 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.631959 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.631949 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-tmp\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.632499 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.632447 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-metrics-client-ca\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.635255 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.635071 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-secret-thanos-querier-tls\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.635483 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.635427 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.635642 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.635608 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.635747 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.635702 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.635908 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.635887 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.636122 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.636107 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-secret-grpc-tls\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.673769 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.673729 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:43.678236 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.678213 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-85b56fd659-pffwm"] Nov 04 04:47:43.680876 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.680860 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:43.688906 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.688886 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2xpb\" (UniqueName: \"kubernetes.io/projected/a06cbcc0-f590-4829-8dc0-1d173b2a4c6e-kube-api-access-l2xpb\") pod \"thanos-querier-7976f76979-72mkk\" (UID: \"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e\") " pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.732909 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.732853 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b97040df-23e0-4ffa-a225-39f4a8c4ae50-console-serving-cert\") pod \"console-85b56fd659-pffwm\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:43.732909 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.732891 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-trusted-ca-bundle\") pod \"console-85b56fd659-pffwm\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:43.732909 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.732909 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvp6w\" (UniqueName: \"kubernetes.io/projected/b97040df-23e0-4ffa-a225-39f4a8c4ae50-kube-api-access-bvp6w\") pod \"console-85b56fd659-pffwm\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:43.733180 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.733070 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b97040df-23e0-4ffa-a225-39f4a8c4ae50-console-oauth-config\") pod \"console-85b56fd659-pffwm\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:43.733180 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.733119 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-oauth-serving-cert\") pod \"console-85b56fd659-pffwm\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:43.733346 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.733204 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-service-ca\") pod \"console-85b56fd659-pffwm\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:43.733346 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.733236 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b97040df-23e0-4ffa-a225-39f4a8c4ae50-tmp\") pod \"console-85b56fd659-pffwm\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:43.733346 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.733264 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-console-config\") pod \"console-85b56fd659-pffwm\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:43.734196 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.734109 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 04 04:47:43.734196 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.734109 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 04 04:47:43.734196 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.734150 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 04 04:47:43.734404 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.734197 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 04 04:47:43.734404 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.734385 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 04 04:47:43.734500 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.734456 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 04 04:47:43.734540 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.734507 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-drn8g\"" Nov 04 04:47:43.734641 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.734620 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 04 04:47:43.734981 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.734967 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 04 04:47:43.740293 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.740276 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85b56fd659-pffwm"] Nov 04 04:47:43.792337 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.792305 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:43.813482 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.813459 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 04:47:43.816899 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:43.816873 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e2ae43_a4a5_4fe0_8310_980d487b0a18.slice/crio-0cbcff27b5a27758e1caf7fb7a579deda0f6ac5450c0509fd31e6b7891eab46e WatchSource:0}: Error finding container 0cbcff27b5a27758e1caf7fb7a579deda0f6ac5450c0509fd31e6b7891eab46e: Status 404 returned error can't find the container with id 0cbcff27b5a27758e1caf7fb7a579deda0f6ac5450c0509fd31e6b7891eab46e Nov 04 04:47:43.834239 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.834212 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-oauth-serving-cert\") pod \"console-85b56fd659-pffwm\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:43.838816 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.834261 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-service-ca\") pod \"console-85b56fd659-pffwm\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:43.838816 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.834293 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b97040df-23e0-4ffa-a225-39f4a8c4ae50-tmp\") pod \"console-85b56fd659-pffwm\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:43.838816 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.834335 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-console-config\") pod \"console-85b56fd659-pffwm\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:43.838816 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.834406 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b97040df-23e0-4ffa-a225-39f4a8c4ae50-console-serving-cert\") pod \"console-85b56fd659-pffwm\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:43.838816 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.834459 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-trusted-ca-bundle\") pod \"console-85b56fd659-pffwm\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:43.838816 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.834482 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvp6w\" (UniqueName: \"kubernetes.io/projected/b97040df-23e0-4ffa-a225-39f4a8c4ae50-kube-api-access-bvp6w\") pod \"console-85b56fd659-pffwm\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:43.838816 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.834533 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b97040df-23e0-4ffa-a225-39f4a8c4ae50-console-oauth-config\") pod \"console-85b56fd659-pffwm\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:43.838816 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.837371 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-service-ca\") pod \"console-85b56fd659-pffwm\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:43.838816 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.837718 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-oauth-serving-cert\") pod \"console-85b56fd659-pffwm\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:43.838816 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.838163 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-console-config\") pod \"console-85b56fd659-pffwm\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:43.838816 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.838361 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-trusted-ca-bundle\") pod \"console-85b56fd659-pffwm\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:43.838816 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.838613 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b97040df-23e0-4ffa-a225-39f4a8c4ae50-tmp\") pod \"console-85b56fd659-pffwm\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:43.839703 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.839684 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b97040df-23e0-4ffa-a225-39f4a8c4ae50-console-serving-cert\") pod \"console-85b56fd659-pffwm\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:43.841505 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.841471 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b97040df-23e0-4ffa-a225-39f4a8c4ae50-console-oauth-config\") pod \"console-85b56fd659-pffwm\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:43.893490 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.893465 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvp6w\" (UniqueName: \"kubernetes.io/projected/b97040df-23e0-4ffa-a225-39f4a8c4ae50-kube-api-access-bvp6w\") pod \"console-85b56fd659-pffwm\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:43.991491 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:43.991392 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:44.171635 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:44.171547 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7976f76979-72mkk"] Nov 04 04:47:44.174905 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:44.174882 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06cbcc0_f590_4829_8dc0_1d173b2a4c6e.slice/crio-51a84019ee7b923a5dcd23b64542a3c9198d1bd61cb469eab1ba781fc7039151 WatchSource:0}: Error finding container 51a84019ee7b923a5dcd23b64542a3c9198d1bd61cb469eab1ba781fc7039151: Status 404 returned error can't find the container with id 51a84019ee7b923a5dcd23b64542a3c9198d1bd61cb469eab1ba781fc7039151 Nov 04 04:47:44.357232 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:44.357191 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2" event={"ID":"a3d46a1d-9630-4f75-97c8-09302e92d701","Type":"ContainerStarted","Data":"8f8a5b13a5883ad444262c71dc7aed1a531610b9d52ea837548d7acf5eeb0073"} Nov 04 04:47:44.357417 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:44.357279 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2" event={"ID":"a3d46a1d-9630-4f75-97c8-09302e92d701","Type":"ContainerStarted","Data":"902c8474b963c992f92d14002f53284ece10ca7aaf39e99c56b8d28309a87c71"} Nov 04 04:47:44.357417 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:44.357309 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2" event={"ID":"a3d46a1d-9630-4f75-97c8-09302e92d701","Type":"ContainerStarted","Data":"e393ab64f02a50c33f959b630d5a3ee6a5ec76826b5d7d63452045f4a993fa76"} Nov 04 04:47:44.359347 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:44.359323 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rpwmg" event={"ID":"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c","Type":"ContainerStarted","Data":"ab0a4c8e2485bd0e72e77e3e0271ed734b660bb30e56a52b971c920aabcea52d"} Nov 04 04:47:44.359450 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:44.359381 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rpwmg" event={"ID":"d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c","Type":"ContainerStarted","Data":"33a68b379a66635ee8c6270aaed1d0440558aafff25bcd93f19651889a8cd369"} Nov 04 04:47:44.360473 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:44.360450 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" event={"ID":"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e","Type":"ContainerStarted","Data":"51a84019ee7b923a5dcd23b64542a3c9198d1bd61cb469eab1ba781fc7039151"} Nov 04 04:47:44.361500 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:44.361481 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5e2ae43-a4a5-4fe0-8310-980d487b0a18","Type":"ContainerStarted","Data":"0cbcff27b5a27758e1caf7fb7a579deda0f6ac5450c0509fd31e6b7891eab46e"} Nov 04 04:47:44.367981 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:44.367960 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85b56fd659-pffwm"] Nov 04 04:47:44.371113 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:44.371088 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97040df_23e0_4ffa_a225_39f4a8c4ae50.slice/crio-047560c3c50b705908b7725465487a33a88a61005fabd9946d0895c5ab45d95b WatchSource:0}: Error finding container 047560c3c50b705908b7725465487a33a88a61005fabd9946d0895c5ab45d95b: Status 404 returned error can't find the container with id 047560c3c50b705908b7725465487a33a88a61005fabd9946d0895c5ab45d95b Nov 04 04:47:44.590620 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:44.590525 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hklc2" podStartSLOduration=2.9349209099999998 podStartE2EDuration="4.590510037s" totalImagesPullingTime="1.655589127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:47:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:44.474454441 +0000 UTC m=+23.365311461" watchObservedRunningTime="2026-11-04 04:47:44.590510037 +0000 UTC m=+23.481367058" Nov 04 04:47:44.590864 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:44.590841 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rpwmg" podStartSLOduration=3.146892291 podStartE2EDuration="4.590835511s" totalImagesPullingTime="1.359589162s" totalInitContainerRuntime="84.354058ms" isStatefulPod=true podCreationTimestamp="2026-11-04 04:47:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:44.590404359 +0000 UTC m=+23.481261383" watchObservedRunningTime="2026-11-04 04:47:44.590835511 +0000 UTC m=+23.481692531" Nov 04 04:47:44.877827 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:44.877791 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:44.998776 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:44.998720 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:45.200609 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.200456 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:45.326625 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.326591 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:45.367222 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.367159 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85b56fd659-pffwm" event={"ID":"b97040df-23e0-4ffa-a225-39f4a8c4ae50","Type":"ContainerStarted","Data":"047560c3c50b705908b7725465487a33a88a61005fabd9946d0895c5ab45d95b"} Nov 04 04:47:45.449529 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.449500 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:45.578972 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.578878 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:45.701193 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.701150 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:45.762822 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.762791 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7f458b6499-5vx8b"] Nov 04 04:47:45.764706 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.764680 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:47:45.818106 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.818075 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 04 04:47:45.818106 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.818094 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 04 04:47:45.818106 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.818117 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 04 04:47:45.818357 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.818162 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xg45s\"" Nov 04 04:47:45.818357 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.818219 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 04 04:47:45.818357 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.818272 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9ibjj0vf3e41l\"" Nov 04 04:47:45.828966 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.828903 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f458b6499-5vx8b"] Nov 04 04:47:45.852505 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.852417 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3e1448fd-a504-43b2-b474-63abb5fc7d17-metrics-server-audit-profiles\") pod \"metrics-server-7f458b6499-5vx8b\" (UID: \"3e1448fd-a504-43b2-b474-63abb5fc7d17\") " pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:47:45.852505 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.852466 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3e1448fd-a504-43b2-b474-63abb5fc7d17-secret-metrics-server-client-certs\") pod \"metrics-server-7f458b6499-5vx8b\" (UID: \"3e1448fd-a504-43b2-b474-63abb5fc7d17\") " pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:47:45.852505 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.852492 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3e1448fd-a504-43b2-b474-63abb5fc7d17-client-ca-bundle\") pod \"metrics-server-7f458b6499-5vx8b\" (UID: \"3e1448fd-a504-43b2-b474-63abb5fc7d17\") " pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:47:45.852766 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.852669 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3e1448fd-a504-43b2-b474-63abb5fc7d17-secret-metrics-server-tls\") pod \"metrics-server-7f458b6499-5vx8b\" (UID: \"3e1448fd-a504-43b2-b474-63abb5fc7d17\") " pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:47:45.852766 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.852709 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3e1448fd-a504-43b2-b474-63abb5fc7d17-audit-log\") pod \"metrics-server-7f458b6499-5vx8b\" (UID: \"3e1448fd-a504-43b2-b474-63abb5fc7d17\") " pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:47:45.852766 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.852751 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e1448fd-a504-43b2-b474-63abb5fc7d17-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f458b6499-5vx8b\" (UID: \"3e1448fd-a504-43b2-b474-63abb5fc7d17\") " pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:47:45.852916 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.852777 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xlbv\" (UniqueName: \"kubernetes.io/projected/3e1448fd-a504-43b2-b474-63abb5fc7d17-kube-api-access-7xlbv\") pod \"metrics-server-7f458b6499-5vx8b\" (UID: \"3e1448fd-a504-43b2-b474-63abb5fc7d17\") " pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:47:45.953947 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.953906 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3e1448fd-a504-43b2-b474-63abb5fc7d17-metrics-server-audit-profiles\") pod \"metrics-server-7f458b6499-5vx8b\" (UID: \"3e1448fd-a504-43b2-b474-63abb5fc7d17\") " pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:47:45.954139 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.953960 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3e1448fd-a504-43b2-b474-63abb5fc7d17-secret-metrics-server-client-certs\") pod \"metrics-server-7f458b6499-5vx8b\" (UID: \"3e1448fd-a504-43b2-b474-63abb5fc7d17\") " pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:47:45.954139 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.954010 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3e1448fd-a504-43b2-b474-63abb5fc7d17-client-ca-bundle\") pod \"metrics-server-7f458b6499-5vx8b\" (UID: \"3e1448fd-a504-43b2-b474-63abb5fc7d17\") " pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:47:45.954139 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.954061 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3e1448fd-a504-43b2-b474-63abb5fc7d17-secret-metrics-server-tls\") pod \"metrics-server-7f458b6499-5vx8b\" (UID: \"3e1448fd-a504-43b2-b474-63abb5fc7d17\") " pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:47:45.954139 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.954092 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3e1448fd-a504-43b2-b474-63abb5fc7d17-audit-log\") pod \"metrics-server-7f458b6499-5vx8b\" (UID: \"3e1448fd-a504-43b2-b474-63abb5fc7d17\") " pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:47:45.954139 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.954128 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e1448fd-a504-43b2-b474-63abb5fc7d17-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f458b6499-5vx8b\" (UID: \"3e1448fd-a504-43b2-b474-63abb5fc7d17\") " pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:47:45.954360 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.954152 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xlbv\" (UniqueName: \"kubernetes.io/projected/3e1448fd-a504-43b2-b474-63abb5fc7d17-kube-api-access-7xlbv\") pod \"metrics-server-7f458b6499-5vx8b\" (UID: \"3e1448fd-a504-43b2-b474-63abb5fc7d17\") " pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:47:45.954962 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.954935 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3e1448fd-a504-43b2-b474-63abb5fc7d17-metrics-server-audit-profiles\") pod \"metrics-server-7f458b6499-5vx8b\" (UID: \"3e1448fd-a504-43b2-b474-63abb5fc7d17\") " pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:47:45.955251 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.955222 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3e1448fd-a504-43b2-b474-63abb5fc7d17-audit-log\") pod \"metrics-server-7f458b6499-5vx8b\" (UID: \"3e1448fd-a504-43b2-b474-63abb5fc7d17\") " pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:47:45.956010 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.955968 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e1448fd-a504-43b2-b474-63abb5fc7d17-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f458b6499-5vx8b\" (UID: \"3e1448fd-a504-43b2-b474-63abb5fc7d17\") " pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:47:45.957053 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.957035 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3e1448fd-a504-43b2-b474-63abb5fc7d17-secret-metrics-server-client-certs\") pod \"metrics-server-7f458b6499-5vx8b\" (UID: \"3e1448fd-a504-43b2-b474-63abb5fc7d17\") " pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:47:45.958017 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.957973 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3e1448fd-a504-43b2-b474-63abb5fc7d17-client-ca-bundle\") pod \"metrics-server-7f458b6499-5vx8b\" (UID: \"3e1448fd-a504-43b2-b474-63abb5fc7d17\") " pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:47:45.958487 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:45.958467 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3e1448fd-a504-43b2-b474-63abb5fc7d17-secret-metrics-server-tls\") pod \"metrics-server-7f458b6499-5vx8b\" (UID: \"3e1448fd-a504-43b2-b474-63abb5fc7d17\") " pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:47:46.012134 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:46.012103 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xlbv\" (UniqueName: \"kubernetes.io/projected/3e1448fd-a504-43b2-b474-63abb5fc7d17-kube-api-access-7xlbv\") pod \"metrics-server-7f458b6499-5vx8b\" (UID: \"3e1448fd-a504-43b2-b474-63abb5fc7d17\") " pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:47:46.076132 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:46.076097 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:47:46.373461 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:46.373426 4581 generic.go:356] "Generic (PLEG): container finished" podID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerID="7b919f421f6e06141ac48be758cc41e77837155c2b66ff2a3117640e40dc4cef" exitCode=0 Nov 04 04:47:46.373622 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:46.373511 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5e2ae43-a4a5-4fe0-8310-980d487b0a18","Type":"ContainerDied","Data":"7b919f421f6e06141ac48be758cc41e77837155c2b66ff2a3117640e40dc4cef"} Nov 04 04:47:46.509368 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:46.509298 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f458b6499-5vx8b"] Nov 04 04:47:46.553043 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:46.553006 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e1448fd_a504_43b2_b474_63abb5fc7d17.slice/crio-9245ab60666df2f4a8dfc2fda0c87229252300501a82dfb6018bb05fec9712b0 WatchSource:0}: Error finding container 9245ab60666df2f4a8dfc2fda0c87229252300501a82dfb6018bb05fec9712b0: Status 404 returned error can't find the container with id 9245ab60666df2f4a8dfc2fda0c87229252300501a82dfb6018bb05fec9712b0 Nov 04 04:47:47.380662 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:47.380621 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" event={"ID":"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e","Type":"ContainerStarted","Data":"3bd9f80a97ad7ea8b1b40309daf72e0914bb8e1a1559476799f7a5a587f0955d"} Nov 04 04:47:47.381152 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:47.380695 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" event={"ID":"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e","Type":"ContainerStarted","Data":"88fbdced74d5bbb64be6da46de22473322b914365370fc9b67bbb812d68d75f8"} Nov 04 04:47:47.381152 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:47.380725 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" event={"ID":"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e","Type":"ContainerStarted","Data":"9fff9fcf285401f8eabff9b8b57c3eb132b3beef3c3a322bcee6175d1d83e9df"} Nov 04 04:47:47.383608 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:47.383579 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" event={"ID":"3e1448fd-a504-43b2-b474-63abb5fc7d17","Type":"ContainerStarted","Data":"9245ab60666df2f4a8dfc2fda0c87229252300501a82dfb6018bb05fec9712b0"} Nov 04 04:47:47.898263 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:47:47.898223 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 04:47:48.391661 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:48.391627 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" event={"ID":"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e","Type":"ContainerStarted","Data":"c8823b4a5004fb1bea6168962d78677c66f9daa6d6280b0fdc105e738c623e21"} Nov 04 04:47:48.393685 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:48.393664 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85b56fd659-pffwm" event={"ID":"b97040df-23e0-4ffa-a225-39f4a8c4ae50","Type":"ContainerStarted","Data":"71294dd3152aaca00def7908ca0a3223b35c40950119c842d2254d616042a656"} Nov 04 04:47:48.509652 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:48.509583 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-85b56fd659-pffwm" podStartSLOduration=2.080414441 podStartE2EDuration="5.509564864s" totalImagesPullingTime="3.429150423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:48.50925485 +0000 UTC m=+27.400111875" watchObservedRunningTime="2026-11-04 04:47:48.509564864 +0000 UTC m=+27.400421885" Nov 04 04:47:49.399418 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.399381 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" event={"ID":"3e1448fd-a504-43b2-b474-63abb5fc7d17","Type":"ContainerStarted","Data":"9ff87e2265965ae341ec07ac9c84095d47fc2e37ca131089799dfdaf4533021f"} Nov 04 04:47:49.402588 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.402560 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" event={"ID":"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e","Type":"ContainerStarted","Data":"0b4cec616edf21de8b6b4907e81710cc91e8b3f68053cda040d09dd33a1b5461"} Nov 04 04:47:49.402722 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.402620 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" event={"ID":"a06cbcc0-f590-4829-8dc0-1d173b2a4c6e","Type":"ContainerStarted","Data":"0b22bf5de56e5983e2d3beb9737c0c3ade1723f82fbaefbf962c7864adb66d28"} Nov 04 04:47:49.402722 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.402670 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:49.405861 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.405836 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5e2ae43-a4a5-4fe0-8310-980d487b0a18","Type":"ContainerStarted","Data":"c55a5d99b452368b1e3bdd3c70be2ad7a305cf2664d04db5f28ef0c7c91129ba"} Nov 04 04:47:49.405978 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.405888 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5e2ae43-a4a5-4fe0-8310-980d487b0a18","Type":"ContainerStarted","Data":"c5c67cf16c03b3feed5097f79c7f907759fa258c2fa81d925360ae7fb17f6334"} Nov 04 04:47:49.405978 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.405912 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5e2ae43-a4a5-4fe0-8310-980d487b0a18","Type":"ContainerStarted","Data":"a272e4db4a1cb059077e053a426305c61d8377df7c98e8b33f382c8a50609901"} Nov 04 04:47:49.405978 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.405945 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5e2ae43-a4a5-4fe0-8310-980d487b0a18","Type":"ContainerStarted","Data":"e7e300a6d6f56c18d1d338cb82f991394118e5022fa2df7009548c393b57d338"} Nov 04 04:47:49.406130 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.406024 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5e2ae43-a4a5-4fe0-8310-980d487b0a18","Type":"ContainerStarted","Data":"15f5032d47b1a82e1daaa7cef532c2b94e03dcf12dc1e931d66cb70df0b974e2"} Nov 04 04:47:49.406130 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.406057 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5e2ae43-a4a5-4fe0-8310-980d487b0a18","Type":"ContainerStarted","Data":"def68bbf0906a1fe8703d43fdba0bdae53c2906a5aa369fc6b6a894f34d6e7c0"} Nov 04 04:47:49.520294 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.520236 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" podStartSLOduration=2.515879771 podStartE2EDuration="4.520218506s" totalImagesPullingTime="2.004338735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:49.519073297 +0000 UTC m=+28.409930318" watchObservedRunningTime="2026-11-04 04:47:49.520218506 +0000 UTC m=+28.411075526" Nov 04 04:47:49.671846 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.671743 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.842717036 podStartE2EDuration="7.67172805s" totalImagesPullingTime="3.811399506s" totalInitContainerRuntime="17.611508ms" isStatefulPod=false podCreationTimestamp="2026-11-04 04:47:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:49.641349411 +0000 UTC m=+28.532206432" watchObservedRunningTime="2026-11-04 04:47:49.67172805 +0000 UTC m=+28.562585108" Nov 04 04:47:49.721437 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.721409 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:49.767510 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.767451 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 04:47:49.768789 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.768731 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" podStartSLOduration=2.841653757 podStartE2EDuration="6.768711822s" totalImagesPullingTime="3.927058065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:47:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:49.766206109 +0000 UTC m=+28.657063132" watchObservedRunningTime="2026-11-04 04:47:49.768711822 +0000 UTC m=+28.659568844" Nov 04 04:47:49.773513 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.773492 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.828106 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.828071 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 04 04:47:49.828106 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.828089 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 04 04:47:49.828106 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.828070 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 04 04:47:49.828106 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.828109 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 04 04:47:49.828421 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.828077 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 04 04:47:49.828421 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.828126 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 04 04:47:49.828421 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.828131 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 04 04:47:49.828421 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.828071 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 04 04:47:49.828421 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.828413 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-pdrn1nm48qj1\"" Nov 04 04:47:49.828634 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.828483 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 04 04:47:49.878570 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.878528 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 04 04:47:49.878570 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.878567 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 04 04:47:49.878771 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.878596 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 04 04:47:49.878771 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.878637 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7h8bv\"" Nov 04 04:47:49.879967 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.879931 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 04 04:47:49.881572 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.881545 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 04:47:49.887761 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.887737 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7e0ecffe-75b6-4a90-a4df-baa32433ce85-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.887862 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.887772 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42tg7\" (UniqueName: \"kubernetes.io/projected/7e0ecffe-75b6-4a90-a4df-baa32433ce85-kube-api-access-42tg7\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.887862 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.887796 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.887862 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.887812 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.888006 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.887864 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-web-config\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.888006 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.887917 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.888076 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.888019 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.888076 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.888059 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e0ecffe-75b6-4a90-a4df-baa32433ce85-config-out\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.888144 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.888095 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.888144 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.888113 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.888144 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.888128 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.888283 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.888166 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.888283 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.888273 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.888375 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.888305 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-config\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.888375 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.888331 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.888445 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.888376 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.888445 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.888394 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.888445 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.888412 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.888532 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.888454 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.888532 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.888506 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.928615 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.928546 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 04 04:47:49.989370 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.989341 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.989532 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.989378 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-config\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.989532 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.989403 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.989532 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.989434 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.989532 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.989457 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.989532 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.989483 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.989532 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.989509 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.989826 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.989536 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.989826 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.989563 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7e0ecffe-75b6-4a90-a4df-baa32433ce85-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.989826 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.989609 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42tg7\" (UniqueName: \"kubernetes.io/projected/7e0ecffe-75b6-4a90-a4df-baa32433ce85-kube-api-access-42tg7\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.989826 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.989651 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.989826 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.989677 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.989826 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.989699 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-web-config\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.989826 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.989724 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.989826 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.989754 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.989826 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.989786 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e0ecffe-75b6-4a90-a4df-baa32433ce85-config-out\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.989826 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.989815 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.990313 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.989840 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.990313 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.989864 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.990313 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.989912 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.990313 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.990106 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.990313 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.990294 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.990542 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.990496 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.990615 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.990584 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.993167 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.992980 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.993787 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.993734 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.994680 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.994633 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.995807 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.995779 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7e0ecffe-75b6-4a90-a4df-baa32433ce85-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.996371 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.996349 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e0ecffe-75b6-4a90-a4df-baa32433ce85-config-out\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:49.996922 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:49.996895 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:50.000957 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:50.000928 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-config\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:50.001121 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:50.001096 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-web-config\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:50.001361 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:50.001334 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:50.001507 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:50.001480 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:50.001618 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:50.001599 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:50.001914 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:50.001894 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:50.002165 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:50.002150 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:50.002362 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:50.002340 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:50.003413 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:50.003394 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:50.046843 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:50.046822 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42tg7\" (UniqueName: \"kubernetes.io/projected/7e0ecffe-75b6-4a90-a4df-baa32433ce85-kube-api-access-42tg7\") pod \"prometheus-k8s-0\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:50.128411 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:50.128389 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:50.642681 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:50.642646 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 04:47:50.646260 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:47:50.646232 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0ecffe_75b6_4a90_a4df_baa32433ce85.slice/crio-13643c64fd2ba8ecd79e5305f724c1d18631a64607f66802ca2be4b960064790 WatchSource:0}: Error finding container 13643c64fd2ba8ecd79e5305f724c1d18631a64607f66802ca2be4b960064790: Status 404 returned error can't find the container with id 13643c64fd2ba8ecd79e5305f724c1d18631a64607f66802ca2be4b960064790 Nov 04 04:47:50.872947 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:50.872908 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:51.047324 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:51.047244 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:51.173453 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:51.173416 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:51.414831 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:51.414799 4581 generic.go:356] "Generic (PLEG): container finished" podID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerID="4cf25f2a6bc820a1aa7ac92d7fdd9514fed80dadbd670eddb97a9f36b87534c4" exitCode=0 Nov 04 04:47:51.415026 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:51.414878 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7e0ecffe-75b6-4a90-a4df-baa32433ce85","Type":"ContainerDied","Data":"4cf25f2a6bc820a1aa7ac92d7fdd9514fed80dadbd670eddb97a9f36b87534c4"} Nov 04 04:47:51.415026 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:51.414945 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7e0ecffe-75b6-4a90-a4df-baa32433ce85","Type":"ContainerStarted","Data":"13643c64fd2ba8ecd79e5305f724c1d18631a64607f66802ca2be4b960064790"} Nov 04 04:47:51.449273 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:51.449243 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:51.741019 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:51.740925 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85b56fd659-pffwm"] Nov 04 04:47:52.060802 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:52.060718 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:52.587780 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:52.587690 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:52.765012 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:52.764955 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:52.907517 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:52.907484 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:53.026765 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:53.026730 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:53.991856 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:53.991823 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:47:54.152958 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:54.152919 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:54.271531 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:54.271351 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:54.815678 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:54.815645 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6xc6w" Nov 04 04:47:55.154390 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:55.154340 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:55.322270 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:55.322241 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:55.414919 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:55.414896 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7976f76979-72mkk" Nov 04 04:47:55.446837 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:55.446804 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:56.312836 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:56.312735 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:56.440692 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:56.440656 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7e0ecffe-75b6-4a90-a4df-baa32433ce85","Type":"ContainerStarted","Data":"22cf1454068603c49f99012546e2657ecdf3686e6e538ccdb765999fb9414538"} Nov 04 04:47:56.440846 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:56.440750 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7e0ecffe-75b6-4a90-a4df-baa32433ce85","Type":"ContainerStarted","Data":"5cbda64fca81e81b23b68f589fd84fac6bb1362093a2fe75b38ec6584489603d"} Nov 04 04:47:56.440846 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:56.440777 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7e0ecffe-75b6-4a90-a4df-baa32433ce85","Type":"ContainerStarted","Data":"1bbf40bb640e17fd2b731d23ce3d840544587bd8c988387e8147e185a6cd6a6e"} Nov 04 04:47:56.440846 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:56.440801 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7e0ecffe-75b6-4a90-a4df-baa32433ce85","Type":"ContainerStarted","Data":"c9ef1308474db7101232fe717122ea21bdb884c36a0fea5ff4f8b50472d8633e"} Nov 04 04:47:56.440846 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:56.440828 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7e0ecffe-75b6-4a90-a4df-baa32433ce85","Type":"ContainerStarted","Data":"397189c442d42f76718561db969fe7e47201d797d4b11da43132620a93503948"} Nov 04 04:47:56.441018 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:56.440852 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7e0ecffe-75b6-4a90-a4df-baa32433ce85","Type":"ContainerStarted","Data":"eae0651623d491be0c19c9324f78475518e6da10aa58ba0c64d34601967e45d6"} Nov 04 04:47:56.512309 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:56.512274 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:56.567759 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:47:56.567679 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.573288484 podStartE2EDuration="7.567661128s" totalImagesPullingTime="3.97820345s" totalInitContainerRuntime="16.169194ms" isStatefulPod=false podCreationTimestamp="2026-11-04 04:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:56.566924093 +0000 UTC m=+35.457781131" watchObservedRunningTime="2026-11-04 04:47:56.567661128 +0000 UTC m=+35.458518149" Nov 04 04:48:00.129105 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:00.129073 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:00.311716 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:00.311684 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:48:00.454791 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:00.454701 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:48:01.609808 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:01.609776 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:01.629622 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:01.629597 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:02.477844 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:02.477819 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:06.076735 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:06.076708 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:48:06.077159 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:06.076765 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:48:07.849932 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:07.849891 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:48:07.988098 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:07.988059 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:48:08.895474 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:08.895437 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:48:09.017861 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:09.017821 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:48:16.752836 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:16.752800 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:48:16.761224 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:16.761192 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-85b56fd659-pffwm" podUID="b97040df-23e0-4ffa-a225-39f4a8c4ae50" containerName="console" containerID="cri-o://71294dd3152aaca00def7908ca0a3223b35c40950119c842d2254d616042a656" gracePeriod=15 Nov 04 04:48:16.874512 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:16.874481 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:48:17.011698 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.011644 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:48:17.133776 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.133747 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-trusted-ca-bundle\") pod \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " Nov 04 04:48:17.133944 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.133799 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b97040df-23e0-4ffa-a225-39f4a8c4ae50-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b97040df-23e0-4ffa-a225-39f4a8c4ae50-tmp\") pod \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " Nov 04 04:48:17.133944 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.133829 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-console-config\" (UniqueName: \"kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-console-config\") pod \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " Nov 04 04:48:17.133944 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.133860 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b97040df-23e0-4ffa-a225-39f4a8c4ae50-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b97040df-23e0-4ffa-a225-39f4a8c4ae50-console-oauth-config\") pod \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " Nov 04 04:48:17.134154 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.133942 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-oauth-serving-cert\") pod \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " Nov 04 04:48:17.134154 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.133982 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b97040df-23e0-4ffa-a225-39f4a8c4ae50-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b97040df-23e0-4ffa-a225-39f4a8c4ae50-console-serving-cert\") pod \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " Nov 04 04:48:17.134154 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.134055 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b97040df-23e0-4ffa-a225-39f4a8c4ae50-kube-api-access-bvp6w\" (UniqueName: \"kubernetes.io/projected/b97040df-23e0-4ffa-a225-39f4a8c4ae50-kube-api-access-bvp6w\") pod \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " Nov 04 04:48:17.134295 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.134172 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-trusted-ca-bundle" pod "b97040df-23e0-4ffa-a225-39f4a8c4ae50" (UID: "b97040df-23e0-4ffa-a225-39f4a8c4ae50"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:48:17.134295 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.134177 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97040df-23e0-4ffa-a225-39f4a8c4ae50-tmp" pod "b97040df-23e0-4ffa-a225-39f4a8c4ae50" (UID: "b97040df-23e0-4ffa-a225-39f4a8c4ae50"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:48:17.134406 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.134236 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-console-config" pod "b97040df-23e0-4ffa-a225-39f4a8c4ae50" (UID: "b97040df-23e0-4ffa-a225-39f4a8c4ae50"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:48:17.134406 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.134349 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-service-ca\" (UniqueName: \"kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-service-ca\") pod \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\" (UID: \"b97040df-23e0-4ffa-a225-39f4a8c4ae50\") " Nov 04 04:48:17.134478 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.134414 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-oauth-serving-cert" pod "b97040df-23e0-4ffa-a225-39f4a8c4ae50" (UID: "b97040df-23e0-4ffa-a225-39f4a8c4ae50"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:48:17.134678 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.134656 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-service-ca" pod "b97040df-23e0-4ffa-a225-39f4a8c4ae50" (UID: "b97040df-23e0-4ffa-a225-39f4a8c4ae50"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:48:17.134678 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.134661 4581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-trusted-ca-bundle\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:17.134977 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.134693 4581 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b97040df-23e0-4ffa-a225-39f4a8c4ae50-tmp\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:17.134977 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.134706 4581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-console-config\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:17.134977 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.134715 4581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-oauth-serving-cert\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:17.136494 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.136471 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b97040df-23e0-4ffa-a225-39f4a8c4ae50-console-serving-cert" pod "b97040df-23e0-4ffa-a225-39f4a8c4ae50" (UID: "b97040df-23e0-4ffa-a225-39f4a8c4ae50"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:48:17.136494 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.136498 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97040df-23e0-4ffa-a225-39f4a8c4ae50-kube-api-access-bvp6w" pod "b97040df-23e0-4ffa-a225-39f4a8c4ae50" (UID: "b97040df-23e0-4ffa-a225-39f4a8c4ae50"). InnerVolumeSpecName "kube-api-access-bvp6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:48:17.136720 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.136534 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b97040df-23e0-4ffa-a225-39f4a8c4ae50-console-oauth-config" pod "b97040df-23e0-4ffa-a225-39f4a8c4ae50" (UID: "b97040df-23e0-4ffa-a225-39f4a8c4ae50"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:48:17.235126 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.235092 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvp6w\" (UniqueName: \"kubernetes.io/projected/b97040df-23e0-4ffa-a225-39f4a8c4ae50-kube-api-access-bvp6w\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:17.235126 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.235121 4581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b97040df-23e0-4ffa-a225-39f4a8c4ae50-service-ca\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:17.235304 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.235135 4581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b97040df-23e0-4ffa-a225-39f4a8c4ae50-console-oauth-config\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:17.235304 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.235145 4581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b97040df-23e0-4ffa-a225-39f4a8c4ae50-console-serving-cert\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:17.518081 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.518048 4581 generic.go:356] "Generic (PLEG): container finished" podID="b97040df-23e0-4ffa-a225-39f4a8c4ae50" containerID="71294dd3152aaca00def7908ca0a3223b35c40950119c842d2254d616042a656" exitCode=0 Nov 04 04:48:17.518277 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.518096 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85b56fd659-pffwm" Nov 04 04:48:17.518277 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.518138 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85b56fd659-pffwm" event={"ID":"b97040df-23e0-4ffa-a225-39f4a8c4ae50","Type":"ContainerDied","Data":"71294dd3152aaca00def7908ca0a3223b35c40950119c842d2254d616042a656"} Nov 04 04:48:17.518277 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.518205 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85b56fd659-pffwm" event={"ID":"b97040df-23e0-4ffa-a225-39f4a8c4ae50","Type":"ContainerDied","Data":"047560c3c50b705908b7725465487a33a88a61005fabd9946d0895c5ab45d95b"} Nov 04 04:48:17.518277 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.518233 4581 scope.go:122] "RemoveContainer" containerID="71294dd3152aaca00def7908ca0a3223b35c40950119c842d2254d616042a656" Nov 04 04:48:17.528030 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.528010 4581 scope.go:122] "RemoveContainer" containerID="71294dd3152aaca00def7908ca0a3223b35c40950119c842d2254d616042a656" Nov 04 04:48:17.528304 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:48:17.528282 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71294dd3152aaca00def7908ca0a3223b35c40950119c842d2254d616042a656\": container with ID starting with 71294dd3152aaca00def7908ca0a3223b35c40950119c842d2254d616042a656 not found: ID does not exist" containerID="71294dd3152aaca00def7908ca0a3223b35c40950119c842d2254d616042a656" Nov 04 04:48:17.528376 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.528315 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71294dd3152aaca00def7908ca0a3223b35c40950119c842d2254d616042a656"} err="failed to get container status \"71294dd3152aaca00def7908ca0a3223b35c40950119c842d2254d616042a656\": rpc error: code = NotFound desc = could not find container \"71294dd3152aaca00def7908ca0a3223b35c40950119c842d2254d616042a656\": container with ID starting with 71294dd3152aaca00def7908ca0a3223b35c40950119c842d2254d616042a656 not found: ID does not exist" Nov 04 04:48:17.686525 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.686392 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85b56fd659-pffwm"] Nov 04 04:48:17.688167 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:17.688142 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85b56fd659-pffwm"] Nov 04 04:48:17.889702 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:48:17.889651 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 04:48:19.228656 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:19.228612 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b97040df-23e0-4ffa-a225-39f4a8c4ae50" path="/var/lib/kubelet/pods/b97040df-23e0-4ffa-a225-39f4a8c4ae50/volumes" Nov 04 04:48:22.855348 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:22.855318 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:48:22.985193 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:22.985163 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:48:26.082163 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:26.082135 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:48:26.085768 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:26.085753 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7f458b6499-5vx8b" Nov 04 04:48:27.552215 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:27.552180 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:48:27.672604 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:27.672564 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:48:30.013409 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:30.013375 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:48:30.135498 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:30.135466 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:48:31.713513 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:31.713466 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:48:31.836477 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:31.836440 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:48:32.091964 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:32.091880 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:48:32.214708 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:32.214671 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:48:35.235056 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:35.235026 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 04:48:35.235523 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:35.235344 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="alertmanager" containerID="cri-o://def68bbf0906a1fe8703d43fdba0bdae53c2906a5aa369fc6b6a894f34d6e7c0" gracePeriod=120 Nov 04 04:48:35.235523 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:35.235372 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="prom-label-proxy" containerID="cri-o://c55a5d99b452368b1e3bdd3c70be2ad7a305cf2664d04db5f28ef0c7c91129ba" gracePeriod=120 Nov 04 04:48:35.235523 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:35.235395 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="kube-rbac-proxy" containerID="cri-o://a272e4db4a1cb059077e053a426305c61d8377df7c98e8b33f382c8a50609901" gracePeriod=120 Nov 04 04:48:35.235523 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:35.235430 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="kube-rbac-proxy-metric" containerID="cri-o://c5c67cf16c03b3feed5097f79c7f907759fa258c2fa81d925360ae7fb17f6334" gracePeriod=120 Nov 04 04:48:35.235523 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:35.235444 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="config-reloader" containerID="cri-o://15f5032d47b1a82e1daaa7cef532c2b94e03dcf12dc1e931d66cb70df0b974e2" gracePeriod=120 Nov 04 04:48:35.235523 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:35.235422 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="kube-rbac-proxy-web" containerID="cri-o://e7e300a6d6f56c18d1d338cb82f991394118e5022fa2df7009548c393b57d338" gracePeriod=120 Nov 04 04:48:35.584221 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:35.584131 4581 generic.go:356] "Generic (PLEG): container finished" podID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerID="c55a5d99b452368b1e3bdd3c70be2ad7a305cf2664d04db5f28ef0c7c91129ba" exitCode=0 Nov 04 04:48:35.584221 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:35.584166 4581 generic.go:356] "Generic (PLEG): container finished" podID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerID="a272e4db4a1cb059077e053a426305c61d8377df7c98e8b33f382c8a50609901" exitCode=0 Nov 04 04:48:35.584221 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:35.584178 4581 generic.go:356] "Generic (PLEG): container finished" podID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerID="15f5032d47b1a82e1daaa7cef532c2b94e03dcf12dc1e931d66cb70df0b974e2" exitCode=0 Nov 04 04:48:35.584221 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:35.584187 4581 generic.go:356] "Generic (PLEG): container finished" podID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerID="def68bbf0906a1fe8703d43fdba0bdae53c2906a5aa369fc6b6a894f34d6e7c0" exitCode=0 Nov 04 04:48:35.584221 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:35.584173 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5e2ae43-a4a5-4fe0-8310-980d487b0a18","Type":"ContainerDied","Data":"c55a5d99b452368b1e3bdd3c70be2ad7a305cf2664d04db5f28ef0c7c91129ba"} Nov 04 04:48:35.584485 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:35.584267 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5e2ae43-a4a5-4fe0-8310-980d487b0a18","Type":"ContainerDied","Data":"a272e4db4a1cb059077e053a426305c61d8377df7c98e8b33f382c8a50609901"} Nov 04 04:48:35.584485 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:35.584295 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5e2ae43-a4a5-4fe0-8310-980d487b0a18","Type":"ContainerDied","Data":"15f5032d47b1a82e1daaa7cef532c2b94e03dcf12dc1e931d66cb70df0b974e2"} Nov 04 04:48:35.584485 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:35.584321 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5e2ae43-a4a5-4fe0-8310-980d487b0a18","Type":"ContainerDied","Data":"def68bbf0906a1fe8703d43fdba0bdae53c2906a5aa369fc6b6a894f34d6e7c0"} Nov 04 04:48:36.481212 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.481190 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:36.591096 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.591019 4581 generic.go:356] "Generic (PLEG): container finished" podID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerID="c5c67cf16c03b3feed5097f79c7f907759fa258c2fa81d925360ae7fb17f6334" exitCode=0 Nov 04 04:48:36.591096 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.591044 4581 generic.go:356] "Generic (PLEG): container finished" podID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerID="e7e300a6d6f56c18d1d338cb82f991394118e5022fa2df7009548c393b57d338" exitCode=0 Nov 04 04:48:36.591286 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.591111 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5e2ae43-a4a5-4fe0-8310-980d487b0a18","Type":"ContainerDied","Data":"c5c67cf16c03b3feed5097f79c7f907759fa258c2fa81d925360ae7fb17f6334"} Nov 04 04:48:36.591286 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.591140 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:36.591286 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.591185 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5e2ae43-a4a5-4fe0-8310-980d487b0a18","Type":"ContainerDied","Data":"e7e300a6d6f56c18d1d338cb82f991394118e5022fa2df7009548c393b57d338"} Nov 04 04:48:36.591286 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.591224 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5e2ae43-a4a5-4fe0-8310-980d487b0a18","Type":"ContainerDied","Data":"0cbcff27b5a27758e1caf7fb7a579deda0f6ac5450c0509fd31e6b7891eab46e"} Nov 04 04:48:36.591286 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.591255 4581 scope.go:122] "RemoveContainer" containerID="c55a5d99b452368b1e3bdd3c70be2ad7a305cf2664d04db5f28ef0c7c91129ba" Nov 04 04:48:36.591577 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.591550 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-tls-assets\" (UniqueName: \"kubernetes.io/projected/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-tls-assets\") pod \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " Nov 04 04:48:36.591635 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.591586 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-alertmanager-main-db\") pod \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " Nov 04 04:48:36.591635 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.591609 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-kube-api-access-89k7r\" (UniqueName: \"kubernetes.io/projected/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-kube-api-access-89k7r\") pod \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " Nov 04 04:48:36.591717 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.591637 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-alertmanager-trusted-ca-bundle\") pod \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " Nov 04 04:48:36.591717 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.591666 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-metrics-client-ca\") pod \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " Nov 04 04:48:36.591717 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.591698 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-main-tls\") pod \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " Nov 04 04:48:36.591850 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.591773 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-web-config\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-web-config\") pod \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " Nov 04 04:48:36.591850 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.591811 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-kube-rbac-proxy-web\") pod \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " Nov 04 04:48:36.591850 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.591843 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " Nov 04 04:48:36.591961 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.591871 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-cluster-tls-config\") pod \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " Nov 04 04:48:36.591961 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.591911 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-alertmanager-main-db" pod "d5e2ae43-a4a5-4fe0-8310-980d487b0a18" (UID: "d5e2ae43-a4a5-4fe0-8310-980d487b0a18"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:48:36.591961 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.591937 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-config-out\") pod \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " Nov 04 04:48:36.592108 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.591967 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-kube-rbac-proxy\") pod \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " Nov 04 04:48:36.592108 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.592031 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-config-volume\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-config-volume\") pod \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\" (UID: \"d5e2ae43-a4a5-4fe0-8310-980d487b0a18\") " Nov 04 04:48:36.592549 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.592273 4581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-alertmanager-main-db\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:36.592549 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.592321 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-alertmanager-trusted-ca-bundle" pod "d5e2ae43-a4a5-4fe0-8310-980d487b0a18" (UID: "d5e2ae43-a4a5-4fe0-8310-980d487b0a18"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:48:36.592549 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.592363 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-metrics-client-ca" pod "d5e2ae43-a4a5-4fe0-8310-980d487b0a18" (UID: "d5e2ae43-a4a5-4fe0-8310-980d487b0a18"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:48:36.595397 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.595363 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-kube-rbac-proxy" pod "d5e2ae43-a4a5-4fe0-8310-980d487b0a18" (UID: "d5e2ae43-a4a5-4fe0-8310-980d487b0a18"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:48:36.595802 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.595771 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-main-tls" pod "d5e2ae43-a4a5-4fe0-8310-980d487b0a18" (UID: "d5e2ae43-a4a5-4fe0-8310-980d487b0a18"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:48:36.596697 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.596674 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-config-out" pod "d5e2ae43-a4a5-4fe0-8310-980d487b0a18" (UID: "d5e2ae43-a4a5-4fe0-8310-980d487b0a18"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:48:36.596783 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.596705 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-config-volume" pod "d5e2ae43-a4a5-4fe0-8310-980d487b0a18" (UID: "d5e2ae43-a4a5-4fe0-8310-980d487b0a18"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:48:36.596783 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.596768 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-kube-api-access-89k7r" pod "d5e2ae43-a4a5-4fe0-8310-980d487b0a18" (UID: "d5e2ae43-a4a5-4fe0-8310-980d487b0a18"). InnerVolumeSpecName "kube-api-access-89k7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:48:36.596883 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.596791 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-tls-assets" pod "d5e2ae43-a4a5-4fe0-8310-980d487b0a18" (UID: "d5e2ae43-a4a5-4fe0-8310-980d487b0a18"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:48:36.596917 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.596896 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-kube-rbac-proxy-web" pod "d5e2ae43-a4a5-4fe0-8310-980d487b0a18" (UID: "d5e2ae43-a4a5-4fe0-8310-980d487b0a18"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:48:36.597229 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.597201 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-kube-rbac-proxy-metric" pod "d5e2ae43-a4a5-4fe0-8310-980d487b0a18" (UID: "d5e2ae43-a4a5-4fe0-8310-980d487b0a18"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:48:36.601730 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.601705 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-cluster-tls-config" pod "d5e2ae43-a4a5-4fe0-8310-980d487b0a18" (UID: "d5e2ae43-a4a5-4fe0-8310-980d487b0a18"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:48:36.606765 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.606746 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-web-config" pod "d5e2ae43-a4a5-4fe0-8310-980d487b0a18" (UID: "d5e2ae43-a4a5-4fe0-8310-980d487b0a18"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:48:36.606817 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.606770 4581 scope.go:122] "RemoveContainer" containerID="c5c67cf16c03b3feed5097f79c7f907759fa258c2fa81d925360ae7fb17f6334" Nov 04 04:48:36.613624 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.613611 4581 scope.go:122] "RemoveContainer" containerID="a272e4db4a1cb059077e053a426305c61d8377df7c98e8b33f382c8a50609901" Nov 04 04:48:36.619978 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.619965 4581 scope.go:122] "RemoveContainer" containerID="e7e300a6d6f56c18d1d338cb82f991394118e5022fa2df7009548c393b57d338" Nov 04 04:48:36.626248 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.626230 4581 scope.go:122] "RemoveContainer" containerID="15f5032d47b1a82e1daaa7cef532c2b94e03dcf12dc1e931d66cb70df0b974e2" Nov 04 04:48:36.632911 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.632895 4581 scope.go:122] "RemoveContainer" containerID="def68bbf0906a1fe8703d43fdba0bdae53c2906a5aa369fc6b6a894f34d6e7c0" Nov 04 04:48:36.639650 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.639630 4581 scope.go:122] "RemoveContainer" containerID="7b919f421f6e06141ac48be758cc41e77837155c2b66ff2a3117640e40dc4cef" Nov 04 04:48:36.648526 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.648508 4581 scope.go:122] "RemoveContainer" containerID="c55a5d99b452368b1e3bdd3c70be2ad7a305cf2664d04db5f28ef0c7c91129ba" Nov 04 04:48:36.648779 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:48:36.648757 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c55a5d99b452368b1e3bdd3c70be2ad7a305cf2664d04db5f28ef0c7c91129ba\": container with ID starting with c55a5d99b452368b1e3bdd3c70be2ad7a305cf2664d04db5f28ef0c7c91129ba not found: ID does not exist" containerID="c55a5d99b452368b1e3bdd3c70be2ad7a305cf2664d04db5f28ef0c7c91129ba" Nov 04 04:48:36.648824 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.648792 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c55a5d99b452368b1e3bdd3c70be2ad7a305cf2664d04db5f28ef0c7c91129ba"} err="failed to get container status \"c55a5d99b452368b1e3bdd3c70be2ad7a305cf2664d04db5f28ef0c7c91129ba\": rpc error: code = NotFound desc = could not find container \"c55a5d99b452368b1e3bdd3c70be2ad7a305cf2664d04db5f28ef0c7c91129ba\": container with ID starting with c55a5d99b452368b1e3bdd3c70be2ad7a305cf2664d04db5f28ef0c7c91129ba not found: ID does not exist" Nov 04 04:48:36.648824 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.648816 4581 scope.go:122] "RemoveContainer" containerID="c5c67cf16c03b3feed5097f79c7f907759fa258c2fa81d925360ae7fb17f6334" Nov 04 04:48:36.649051 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:48:36.649036 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5c67cf16c03b3feed5097f79c7f907759fa258c2fa81d925360ae7fb17f6334\": container with ID starting with c5c67cf16c03b3feed5097f79c7f907759fa258c2fa81d925360ae7fb17f6334 not found: ID does not exist" containerID="c5c67cf16c03b3feed5097f79c7f907759fa258c2fa81d925360ae7fb17f6334" Nov 04 04:48:36.649097 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.649056 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c67cf16c03b3feed5097f79c7f907759fa258c2fa81d925360ae7fb17f6334"} err="failed to get container status \"c5c67cf16c03b3feed5097f79c7f907759fa258c2fa81d925360ae7fb17f6334\": rpc error: code = NotFound desc = could not find container \"c5c67cf16c03b3feed5097f79c7f907759fa258c2fa81d925360ae7fb17f6334\": container with ID starting with c5c67cf16c03b3feed5097f79c7f907759fa258c2fa81d925360ae7fb17f6334 not found: ID does not exist" Nov 04 04:48:36.649097 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.649069 4581 scope.go:122] "RemoveContainer" containerID="a272e4db4a1cb059077e053a426305c61d8377df7c98e8b33f382c8a50609901" Nov 04 04:48:36.649279 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:48:36.649260 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a272e4db4a1cb059077e053a426305c61d8377df7c98e8b33f382c8a50609901\": container with ID starting with a272e4db4a1cb059077e053a426305c61d8377df7c98e8b33f382c8a50609901 not found: ID does not exist" containerID="a272e4db4a1cb059077e053a426305c61d8377df7c98e8b33f382c8a50609901" Nov 04 04:48:36.649320 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.649283 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a272e4db4a1cb059077e053a426305c61d8377df7c98e8b33f382c8a50609901"} err="failed to get container status \"a272e4db4a1cb059077e053a426305c61d8377df7c98e8b33f382c8a50609901\": rpc error: code = NotFound desc = could not find container \"a272e4db4a1cb059077e053a426305c61d8377df7c98e8b33f382c8a50609901\": container with ID starting with a272e4db4a1cb059077e053a426305c61d8377df7c98e8b33f382c8a50609901 not found: ID does not exist" Nov 04 04:48:36.649320 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.649296 4581 scope.go:122] "RemoveContainer" containerID="e7e300a6d6f56c18d1d338cb82f991394118e5022fa2df7009548c393b57d338" Nov 04 04:48:36.649497 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:48:36.649480 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7e300a6d6f56c18d1d338cb82f991394118e5022fa2df7009548c393b57d338\": container with ID starting with e7e300a6d6f56c18d1d338cb82f991394118e5022fa2df7009548c393b57d338 not found: ID does not exist" containerID="e7e300a6d6f56c18d1d338cb82f991394118e5022fa2df7009548c393b57d338" Nov 04 04:48:36.649537 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.649505 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7e300a6d6f56c18d1d338cb82f991394118e5022fa2df7009548c393b57d338"} err="failed to get container status \"e7e300a6d6f56c18d1d338cb82f991394118e5022fa2df7009548c393b57d338\": rpc error: code = NotFound desc = could not find container \"e7e300a6d6f56c18d1d338cb82f991394118e5022fa2df7009548c393b57d338\": container with ID starting with e7e300a6d6f56c18d1d338cb82f991394118e5022fa2df7009548c393b57d338 not found: ID does not exist" Nov 04 04:48:36.649537 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.649525 4581 scope.go:122] "RemoveContainer" containerID="15f5032d47b1a82e1daaa7cef532c2b94e03dcf12dc1e931d66cb70df0b974e2" Nov 04 04:48:36.649760 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:48:36.649746 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15f5032d47b1a82e1daaa7cef532c2b94e03dcf12dc1e931d66cb70df0b974e2\": container with ID starting with 15f5032d47b1a82e1daaa7cef532c2b94e03dcf12dc1e931d66cb70df0b974e2 not found: ID does not exist" containerID="15f5032d47b1a82e1daaa7cef532c2b94e03dcf12dc1e931d66cb70df0b974e2" Nov 04 04:48:36.649799 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.649762 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15f5032d47b1a82e1daaa7cef532c2b94e03dcf12dc1e931d66cb70df0b974e2"} err="failed to get container status \"15f5032d47b1a82e1daaa7cef532c2b94e03dcf12dc1e931d66cb70df0b974e2\": rpc error: code = NotFound desc = could not find container \"15f5032d47b1a82e1daaa7cef532c2b94e03dcf12dc1e931d66cb70df0b974e2\": container with ID starting with 15f5032d47b1a82e1daaa7cef532c2b94e03dcf12dc1e931d66cb70df0b974e2 not found: ID does not exist" Nov 04 04:48:36.649799 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.649774 4581 scope.go:122] "RemoveContainer" containerID="def68bbf0906a1fe8703d43fdba0bdae53c2906a5aa369fc6b6a894f34d6e7c0" Nov 04 04:48:36.649966 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:48:36.649952 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"def68bbf0906a1fe8703d43fdba0bdae53c2906a5aa369fc6b6a894f34d6e7c0\": container with ID starting with def68bbf0906a1fe8703d43fdba0bdae53c2906a5aa369fc6b6a894f34d6e7c0 not found: ID does not exist" containerID="def68bbf0906a1fe8703d43fdba0bdae53c2906a5aa369fc6b6a894f34d6e7c0" Nov 04 04:48:36.650031 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.649969 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"def68bbf0906a1fe8703d43fdba0bdae53c2906a5aa369fc6b6a894f34d6e7c0"} err="failed to get container status \"def68bbf0906a1fe8703d43fdba0bdae53c2906a5aa369fc6b6a894f34d6e7c0\": rpc error: code = NotFound desc = could not find container \"def68bbf0906a1fe8703d43fdba0bdae53c2906a5aa369fc6b6a894f34d6e7c0\": container with ID starting with def68bbf0906a1fe8703d43fdba0bdae53c2906a5aa369fc6b6a894f34d6e7c0 not found: ID does not exist" Nov 04 04:48:36.650031 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.649980 4581 scope.go:122] "RemoveContainer" containerID="7b919f421f6e06141ac48be758cc41e77837155c2b66ff2a3117640e40dc4cef" Nov 04 04:48:36.650208 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:48:36.650193 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b919f421f6e06141ac48be758cc41e77837155c2b66ff2a3117640e40dc4cef\": container with ID starting with 7b919f421f6e06141ac48be758cc41e77837155c2b66ff2a3117640e40dc4cef not found: ID does not exist" containerID="7b919f421f6e06141ac48be758cc41e77837155c2b66ff2a3117640e40dc4cef" Nov 04 04:48:36.650246 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.650211 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b919f421f6e06141ac48be758cc41e77837155c2b66ff2a3117640e40dc4cef"} err="failed to get container status \"7b919f421f6e06141ac48be758cc41e77837155c2b66ff2a3117640e40dc4cef\": rpc error: code = NotFound desc = could not find container \"7b919f421f6e06141ac48be758cc41e77837155c2b66ff2a3117640e40dc4cef\": container with ID starting with 7b919f421f6e06141ac48be758cc41e77837155c2b66ff2a3117640e40dc4cef not found: ID does not exist" Nov 04 04:48:36.650246 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.650222 4581 scope.go:122] "RemoveContainer" containerID="c55a5d99b452368b1e3bdd3c70be2ad7a305cf2664d04db5f28ef0c7c91129ba" Nov 04 04:48:36.650455 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.650439 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c55a5d99b452368b1e3bdd3c70be2ad7a305cf2664d04db5f28ef0c7c91129ba"} err="failed to get container status \"c55a5d99b452368b1e3bdd3c70be2ad7a305cf2664d04db5f28ef0c7c91129ba\": rpc error: code = NotFound desc = could not find container \"c55a5d99b452368b1e3bdd3c70be2ad7a305cf2664d04db5f28ef0c7c91129ba\": container with ID starting with c55a5d99b452368b1e3bdd3c70be2ad7a305cf2664d04db5f28ef0c7c91129ba not found: ID does not exist" Nov 04 04:48:36.650455 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.650454 4581 scope.go:122] "RemoveContainer" containerID="c5c67cf16c03b3feed5097f79c7f907759fa258c2fa81d925360ae7fb17f6334" Nov 04 04:48:36.650640 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.650624 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c67cf16c03b3feed5097f79c7f907759fa258c2fa81d925360ae7fb17f6334"} err="failed to get container status \"c5c67cf16c03b3feed5097f79c7f907759fa258c2fa81d925360ae7fb17f6334\": rpc error: code = NotFound desc = could not find container \"c5c67cf16c03b3feed5097f79c7f907759fa258c2fa81d925360ae7fb17f6334\": container with ID starting with c5c67cf16c03b3feed5097f79c7f907759fa258c2fa81d925360ae7fb17f6334 not found: ID does not exist" Nov 04 04:48:36.650681 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.650640 4581 scope.go:122] "RemoveContainer" containerID="a272e4db4a1cb059077e053a426305c61d8377df7c98e8b33f382c8a50609901" Nov 04 04:48:36.650823 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.650808 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a272e4db4a1cb059077e053a426305c61d8377df7c98e8b33f382c8a50609901"} err="failed to get container status \"a272e4db4a1cb059077e053a426305c61d8377df7c98e8b33f382c8a50609901\": rpc error: code = NotFound desc = could not find container \"a272e4db4a1cb059077e053a426305c61d8377df7c98e8b33f382c8a50609901\": container with ID starting with a272e4db4a1cb059077e053a426305c61d8377df7c98e8b33f382c8a50609901 not found: ID does not exist" Nov 04 04:48:36.650866 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.650823 4581 scope.go:122] "RemoveContainer" containerID="e7e300a6d6f56c18d1d338cb82f991394118e5022fa2df7009548c393b57d338" Nov 04 04:48:36.651061 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.650975 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7e300a6d6f56c18d1d338cb82f991394118e5022fa2df7009548c393b57d338"} err="failed to get container status \"e7e300a6d6f56c18d1d338cb82f991394118e5022fa2df7009548c393b57d338\": rpc error: code = NotFound desc = could not find container \"e7e300a6d6f56c18d1d338cb82f991394118e5022fa2df7009548c393b57d338\": container with ID starting with e7e300a6d6f56c18d1d338cb82f991394118e5022fa2df7009548c393b57d338 not found: ID does not exist" Nov 04 04:48:36.651115 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.651062 4581 scope.go:122] "RemoveContainer" containerID="15f5032d47b1a82e1daaa7cef532c2b94e03dcf12dc1e931d66cb70df0b974e2" Nov 04 04:48:36.651249 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.651233 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15f5032d47b1a82e1daaa7cef532c2b94e03dcf12dc1e931d66cb70df0b974e2"} err="failed to get container status \"15f5032d47b1a82e1daaa7cef532c2b94e03dcf12dc1e931d66cb70df0b974e2\": rpc error: code = NotFound desc = could not find container \"15f5032d47b1a82e1daaa7cef532c2b94e03dcf12dc1e931d66cb70df0b974e2\": container with ID starting with 15f5032d47b1a82e1daaa7cef532c2b94e03dcf12dc1e931d66cb70df0b974e2 not found: ID does not exist" Nov 04 04:48:36.651289 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.651249 4581 scope.go:122] "RemoveContainer" containerID="def68bbf0906a1fe8703d43fdba0bdae53c2906a5aa369fc6b6a894f34d6e7c0" Nov 04 04:48:36.651428 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.651410 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"def68bbf0906a1fe8703d43fdba0bdae53c2906a5aa369fc6b6a894f34d6e7c0"} err="failed to get container status \"def68bbf0906a1fe8703d43fdba0bdae53c2906a5aa369fc6b6a894f34d6e7c0\": rpc error: code = NotFound desc = could not find container \"def68bbf0906a1fe8703d43fdba0bdae53c2906a5aa369fc6b6a894f34d6e7c0\": container with ID starting with def68bbf0906a1fe8703d43fdba0bdae53c2906a5aa369fc6b6a894f34d6e7c0 not found: ID does not exist" Nov 04 04:48:36.651470 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.651431 4581 scope.go:122] "RemoveContainer" containerID="7b919f421f6e06141ac48be758cc41e77837155c2b66ff2a3117640e40dc4cef" Nov 04 04:48:36.651616 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.651601 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b919f421f6e06141ac48be758cc41e77837155c2b66ff2a3117640e40dc4cef"} err="failed to get container status \"7b919f421f6e06141ac48be758cc41e77837155c2b66ff2a3117640e40dc4cef\": rpc error: code = NotFound desc = could not find container \"7b919f421f6e06141ac48be758cc41e77837155c2b66ff2a3117640e40dc4cef\": container with ID starting with 7b919f421f6e06141ac48be758cc41e77837155c2b66ff2a3117640e40dc4cef not found: ID does not exist" Nov 04 04:48:36.693618 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.693596 4581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-config-out\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:36.693618 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.693618 4581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:36.693742 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.693628 4581 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-config-volume\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:36.693742 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.693638 4581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-tls-assets\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:36.693742 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.693647 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89k7r\" (UniqueName: \"kubernetes.io/projected/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-kube-api-access-89k7r\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:36.693742 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.693657 4581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:36.693742 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.693665 4581 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-metrics-client-ca\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:36.693742 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.693674 4581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-main-tls\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:36.693742 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.693683 4581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-web-config\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:36.693742 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.693692 4581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:36.693742 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.693701 4581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:36.693742 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:36.693710 4581 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d5e2ae43-a4a5-4fe0-8310-980d487b0a18-cluster-tls-config\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:37.014703 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.014670 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 04:48:37.017624 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.017596 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 04:48:37.042131 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.042104 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 04:48:37.042414 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.042399 4581 cpu_manager.go:414] "containerMap: removing container" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="kube-rbac-proxy" Nov 04 04:48:37.042486 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.042417 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="kube-rbac-proxy" Nov 04 04:48:37.042486 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.042435 4581 cpu_manager.go:414] "containerMap: removing container" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="init-config-reloader" Nov 04 04:48:37.042486 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.042442 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="init-config-reloader" Nov 04 04:48:37.042486 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.042455 4581 cpu_manager.go:414] "containerMap: removing container" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="alertmanager" Nov 04 04:48:37.042486 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.042463 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="alertmanager" Nov 04 04:48:37.042486 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.042476 4581 cpu_manager.go:414] "containerMap: removing container" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="kube-rbac-proxy-metric" Nov 04 04:48:37.042486 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.042483 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="kube-rbac-proxy-metric" Nov 04 04:48:37.042831 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.042496 4581 cpu_manager.go:414] "containerMap: removing container" podUID="b97040df-23e0-4ffa-a225-39f4a8c4ae50" containerName="console" Nov 04 04:48:37.042831 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.042503 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b97040df-23e0-4ffa-a225-39f4a8c4ae50" containerName="console" Nov 04 04:48:37.042831 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.042537 4581 cpu_manager.go:414] "containerMap: removing container" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="kube-rbac-proxy-web" Nov 04 04:48:37.042831 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.042544 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="kube-rbac-proxy-web" Nov 04 04:48:37.042831 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.042556 4581 cpu_manager.go:414] "containerMap: removing container" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="prom-label-proxy" Nov 04 04:48:37.042831 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.042563 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="prom-label-proxy" Nov 04 04:48:37.042831 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.042575 4581 cpu_manager.go:414] "containerMap: removing container" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="config-reloader" Nov 04 04:48:37.042831 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.042584 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="config-reloader" Nov 04 04:48:37.042831 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.042646 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="prom-label-proxy" Nov 04 04:48:37.042831 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.042657 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="config-reloader" Nov 04 04:48:37.042831 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.042667 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="kube-rbac-proxy" Nov 04 04:48:37.042831 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.042678 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="alertmanager" Nov 04 04:48:37.042831 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.042687 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="kube-rbac-proxy-metric" Nov 04 04:48:37.042831 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.042695 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="b97040df-23e0-4ffa-a225-39f4a8c4ae50" containerName="console" Nov 04 04:48:37.042831 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.042708 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" containerName="kube-rbac-proxy-web" Nov 04 04:48:37.046492 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.046473 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.097234 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.097196 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.097234 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.097231 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.099473 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.097247 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-web-config\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.099473 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.097268 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.099473 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.097408 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-config-volume\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.099473 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.097444 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.099473 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.097473 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.099473 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.097517 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.099473 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.097545 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.099473 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.097570 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-config-out\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.099473 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.097605 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.099473 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.097631 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.099473 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.097665 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrp7j\" (UniqueName: \"kubernetes.io/projected/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-kube-api-access-nrp7j\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.099968 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.099746 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 04 04:48:37.099968 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.099766 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 04 04:48:37.099968 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.099746 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 04 04:48:37.099968 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.099945 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 04 04:48:37.100291 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.099974 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 04 04:48:37.100291 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.100015 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 04 04:48:37.100291 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.100041 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 04 04:48:37.100291 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.100040 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 04 04:48:37.100291 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.100092 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-chrbj\"" Nov 04 04:48:37.100718 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.100703 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 04 04:48:37.150818 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.150757 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 04:48:37.198709 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.198679 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.198854 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.198716 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.198854 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.198736 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-web-config\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.198854 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.198785 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.198854 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.198819 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-config-volume\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.198854 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.198845 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.199114 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.198861 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.199114 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.198897 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.199114 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.198924 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.199114 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.198951 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-config-out\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.199114 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.199020 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.199114 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.199045 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.199114 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.199079 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrp7j\" (UniqueName: \"kubernetes.io/projected/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-kube-api-access-nrp7j\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.199850 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.199779 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.200525 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.200241 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.200525 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.200368 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.202547 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.202296 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-config-volume\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.202547 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.202415 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.202547 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.202429 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.202547 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.202497 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.202774 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.202649 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-web-config\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.202774 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.202665 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.202948 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.202924 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.203605 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.203589 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-config-out\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.203848 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.203834 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.228370 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.228347 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5e2ae43-a4a5-4fe0-8310-980d487b0a18" path="/var/lib/kubelet/pods/d5e2ae43-a4a5-4fe0-8310-980d487b0a18/volumes" Nov 04 04:48:37.256692 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.256670 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrp7j\" (UniqueName: \"kubernetes.io/projected/5c11b727-b2d3-4c3e-ba45-4335b8af0d2d-kube-api-access-nrp7j\") pod \"alertmanager-main-0\" (UID: \"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.363026 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.362978 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 04 04:48:37.786959 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.786810 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 04:48:37.789428 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:48:37.789395 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c11b727_b2d3_4c3e_ba45_4335b8af0d2d.slice/crio-be809ea804a3860578768e756c5f5475ccad495154081ed0087a0d89c14b406f WatchSource:0}: Error finding container be809ea804a3860578768e756c5f5475ccad495154081ed0087a0d89c14b406f: Status 404 returned error can't find the container with id be809ea804a3860578768e756c5f5475ccad495154081ed0087a0d89c14b406f Nov 04 04:48:37.855886 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:37.855858 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:48:38.028814 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:38.028779 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:48:38.600390 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:38.600355 4581 generic.go:356] "Generic (PLEG): container finished" podID="5c11b727-b2d3-4c3e-ba45-4335b8af0d2d" containerID="3b207cd98c37130d4c23e830e4f64cbaa4ac29f65a82de2434a3e12838eb8b4d" exitCode=0 Nov 04 04:48:38.600588 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:38.600438 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d","Type":"ContainerDied","Data":"3b207cd98c37130d4c23e830e4f64cbaa4ac29f65a82de2434a3e12838eb8b4d"} Nov 04 04:48:38.600588 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:38.600508 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d","Type":"ContainerStarted","Data":"be809ea804a3860578768e756c5f5475ccad495154081ed0087a0d89c14b406f"} Nov 04 04:48:39.607616 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:39.607582 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d","Type":"ContainerStarted","Data":"cb6bbe55a6e3235bbf056e620173a476e586505bde5751d9dff447d6b0c0bff0"} Nov 04 04:48:39.607969 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:39.607652 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d","Type":"ContainerStarted","Data":"1ec9db9ca4b7b942e8559b6bcdcf48cd670fbf97d92018f3ca39a3080170ee12"} Nov 04 04:48:39.607969 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:39.607679 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d","Type":"ContainerStarted","Data":"dc17ffec483a22786b8289992504036fd494b43d66fe1f59bbceab452a85e229"} Nov 04 04:48:39.607969 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:39.607704 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d","Type":"ContainerStarted","Data":"34404ccad35bad32c30dab2a1bec35983285a25170e011887933870e8de21418"} Nov 04 04:48:39.607969 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:39.607732 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d","Type":"ContainerStarted","Data":"17874494ee0daa1ea43d02bde22d3fcffec4270760ac156a34f4e78a55c2c7ce"} Nov 04 04:48:39.607969 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:39.607756 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c11b727-b2d3-4c3e-ba45-4335b8af0d2d","Type":"ContainerStarted","Data":"2e6f8a2f3ff461ad19c7b4a37d53604453f55f00a5c4ca8df3e401327f781642"} Nov 04 04:48:39.733033 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:39.732966 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.718853917 podStartE2EDuration="3.732952711s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.098794ms" isStatefulPod=false podCreationTimestamp="2026-11-04 04:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:48:39.732067041 +0000 UTC m=+78.622924074" watchObservedRunningTime="2026-11-04 04:48:39.732952711 +0000 UTC m=+78.623809730" Nov 04 04:48:40.971219 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:40.971181 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 04:48:40.971659 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:40.971519 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="prometheus" containerID="cri-o://eae0651623d491be0c19c9324f78475518e6da10aa58ba0c64d34601967e45d6" gracePeriod=600 Nov 04 04:48:40.971743 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:40.971685 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="kube-rbac-proxy-thanos" containerID="cri-o://22cf1454068603c49f99012546e2657ecdf3686e6e538ccdb765999fb9414538" gracePeriod=600 Nov 04 04:48:40.971798 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:40.971741 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="thanos-sidecar" containerID="cri-o://c9ef1308474db7101232fe717122ea21bdb884c36a0fea5ff4f8b50472d8633e" gracePeriod=600 Nov 04 04:48:40.971798 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:40.971777 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="config-reloader" containerID="cri-o://397189c442d42f76718561db969fe7e47201d797d4b11da43132620a93503948" gracePeriod=600 Nov 04 04:48:40.971899 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:40.971818 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="kube-rbac-proxy" containerID="cri-o://5cbda64fca81e81b23b68f589fd84fac6bb1362093a2fe75b38ec6584489603d" gracePeriod=600 Nov 04 04:48:40.971899 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:40.971881 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="kube-rbac-proxy-web" containerID="cri-o://1bbf40bb640e17fd2b731d23ce3d840544587bd8c988387e8147e185a6cd6a6e" gracePeriod=600 Nov 04 04:48:41.619517 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:41.619481 4581 generic.go:356] "Generic (PLEG): container finished" podID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerID="22cf1454068603c49f99012546e2657ecdf3686e6e538ccdb765999fb9414538" exitCode=0 Nov 04 04:48:41.619517 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:41.619510 4581 generic.go:356] "Generic (PLEG): container finished" podID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerID="5cbda64fca81e81b23b68f589fd84fac6bb1362093a2fe75b38ec6584489603d" exitCode=0 Nov 04 04:48:41.619517 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:41.619518 4581 generic.go:356] "Generic (PLEG): container finished" podID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerID="c9ef1308474db7101232fe717122ea21bdb884c36a0fea5ff4f8b50472d8633e" exitCode=0 Nov 04 04:48:41.619517 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:41.619526 4581 generic.go:356] "Generic (PLEG): container finished" podID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerID="397189c442d42f76718561db969fe7e47201d797d4b11da43132620a93503948" exitCode=0 Nov 04 04:48:41.619783 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:41.619535 4581 generic.go:356] "Generic (PLEG): container finished" podID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerID="eae0651623d491be0c19c9324f78475518e6da10aa58ba0c64d34601967e45d6" exitCode=0 Nov 04 04:48:41.619783 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:41.619551 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7e0ecffe-75b6-4a90-a4df-baa32433ce85","Type":"ContainerDied","Data":"22cf1454068603c49f99012546e2657ecdf3686e6e538ccdb765999fb9414538"} Nov 04 04:48:41.619783 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:41.619646 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7e0ecffe-75b6-4a90-a4df-baa32433ce85","Type":"ContainerDied","Data":"5cbda64fca81e81b23b68f589fd84fac6bb1362093a2fe75b38ec6584489603d"} Nov 04 04:48:41.619783 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:41.619674 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7e0ecffe-75b6-4a90-a4df-baa32433ce85","Type":"ContainerDied","Data":"c9ef1308474db7101232fe717122ea21bdb884c36a0fea5ff4f8b50472d8633e"} Nov 04 04:48:41.619783 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:41.619700 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7e0ecffe-75b6-4a90-a4df-baa32433ce85","Type":"ContainerDied","Data":"397189c442d42f76718561db969fe7e47201d797d4b11da43132620a93503948"} Nov 04 04:48:41.619783 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:41.619723 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7e0ecffe-75b6-4a90-a4df-baa32433ce85","Type":"ContainerDied","Data":"eae0651623d491be0c19c9324f78475518e6da10aa58ba0c64d34601967e45d6"} Nov 04 04:48:42.220329 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.220308 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:42.341734 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.341660 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-prometheus-k8s-tls\") pod \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " Nov 04 04:48:42.341734 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.341704 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-thanos-prometheus-http-client-file\") pod \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " Nov 04 04:48:42.341908 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.341739 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-config\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-config\") pod \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " Nov 04 04:48:42.341908 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.341828 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-configmap-serving-certs-ca-bundle\") pod \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " Nov 04 04:48:42.341908 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.341872 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-rulefiles-0\") pod \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " Nov 04 04:48:42.341908 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.341903 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0ecffe-75b6-4a90-a4df-baa32433ce85-tls-assets\" (UniqueName: \"kubernetes.io/projected/7e0ecffe-75b6-4a90-a4df-baa32433ce85-tls-assets\") pod \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " Nov 04 04:48:42.342133 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.341929 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-web-config\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-web-config\") pod \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " Nov 04 04:48:42.342133 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.342006 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-rulefiles-2\") pod \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " Nov 04 04:48:42.342133 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.342056 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0ecffe-75b6-4a90-a4df-baa32433ce85-config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e0ecffe-75b6-4a90-a4df-baa32433ce85-config-out\") pod \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " Nov 04 04:48:42.342133 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.342083 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-kube-rbac-proxy\") pod \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " Nov 04 04:48:42.342133 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.342118 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-db\") pod \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " Nov 04 04:48:42.342394 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.342157 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-rulefiles-1\") pod \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " Nov 04 04:48:42.342394 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.342189 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " Nov 04 04:48:42.342394 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.342218 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-configmap-kubelet-serving-ca-bundle\") pod \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " Nov 04 04:48:42.342394 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.342234 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-configmap-serving-certs-ca-bundle" pod "7e0ecffe-75b6-4a90-a4df-baa32433ce85" (UID: "7e0ecffe-75b6-4a90-a4df-baa32433ce85"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:48:42.342394 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.342255 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-metrics-client-certs\") pod \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " Nov 04 04:48:42.342394 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.342299 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0ecffe-75b6-4a90-a4df-baa32433ce85-kube-api-access-42tg7\" (UniqueName: \"kubernetes.io/projected/7e0ecffe-75b6-4a90-a4df-baa32433ce85-kube-api-access-42tg7\") pod \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " Nov 04 04:48:42.342394 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.342326 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-grpc-tls\") pod \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " Nov 04 04:48:42.342394 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.342365 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " Nov 04 04:48:42.342394 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.342393 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-configmap-metrics-client-ca\") pod \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " Nov 04 04:48:42.342818 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.342430 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-trusted-ca-bundle\") pod \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\" (UID: \"7e0ecffe-75b6-4a90-a4df-baa32433ce85\") " Nov 04 04:48:42.343296 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.342954 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-rulefiles-1" pod "7e0ecffe-75b6-4a90-a4df-baa32433ce85" (UID: "7e0ecffe-75b6-4a90-a4df-baa32433ce85"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:48:42.343415 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.343374 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-configmap-kubelet-serving-ca-bundle" pod "7e0ecffe-75b6-4a90-a4df-baa32433ce85" (UID: "7e0ecffe-75b6-4a90-a4df-baa32433ce85"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:48:42.343415 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.343392 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-rulefiles-2" pod "7e0ecffe-75b6-4a90-a4df-baa32433ce85" (UID: "7e0ecffe-75b6-4a90-a4df-baa32433ce85"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:48:42.343932 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.343908 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-configmap-metrics-client-ca" pod "7e0ecffe-75b6-4a90-a4df-baa32433ce85" (UID: "7e0ecffe-75b6-4a90-a4df-baa32433ce85"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:48:42.343932 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.343914 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-trusted-ca-bundle" pod "7e0ecffe-75b6-4a90-a4df-baa32433ce85" (UID: "7e0ecffe-75b6-4a90-a4df-baa32433ce85"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:48:42.344452 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.344403 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-rulefiles-0" pod "7e0ecffe-75b6-4a90-a4df-baa32433ce85" (UID: "7e0ecffe-75b6-4a90-a4df-baa32433ce85"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:48:42.345108 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.344920 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-config" pod "7e0ecffe-75b6-4a90-a4df-baa32433ce85" (UID: "7e0ecffe-75b6-4a90-a4df-baa32433ce85"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:48:42.345108 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.344939 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-db" pod "7e0ecffe-75b6-4a90-a4df-baa32433ce85" (UID: "7e0ecffe-75b6-4a90-a4df-baa32433ce85"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:48:42.345108 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.344946 4581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:42.345108 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.344970 4581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:42.345108 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.345012 4581 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:42.345108 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.345029 4581 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-configmap-metrics-client-ca\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:42.345108 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.345042 4581 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:42.345108 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.345056 4581 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:42.345108 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.345070 4581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:42.345739 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.345159 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-thanos-prometheus-http-client-file" pod "7e0ecffe-75b6-4a90-a4df-baa32433ce85" (UID: "7e0ecffe-75b6-4a90-a4df-baa32433ce85"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:48:42.346839 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.346810 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-prometheus-k8s-tls" pod "7e0ecffe-75b6-4a90-a4df-baa32433ce85" (UID: "7e0ecffe-75b6-4a90-a4df-baa32433ce85"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:48:42.347218 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.347191 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0ecffe-75b6-4a90-a4df-baa32433ce85-config-out" pod "7e0ecffe-75b6-4a90-a4df-baa32433ce85" (UID: "7e0ecffe-75b6-4a90-a4df-baa32433ce85"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:48:42.347296 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.347235 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0ecffe-75b6-4a90-a4df-baa32433ce85-tls-assets" pod "7e0ecffe-75b6-4a90-a4df-baa32433ce85" (UID: "7e0ecffe-75b6-4a90-a4df-baa32433ce85"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:48:42.347535 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.347484 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-grpc-tls" pod "7e0ecffe-75b6-4a90-a4df-baa32433ce85" (UID: "7e0ecffe-75b6-4a90-a4df-baa32433ce85"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:48:42.347535 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.347524 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-prometheus-k8s-thanos-sidecar-tls" pod "7e0ecffe-75b6-4a90-a4df-baa32433ce85" (UID: "7e0ecffe-75b6-4a90-a4df-baa32433ce85"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:48:42.347793 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.347770 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-prometheus-k8s-kube-rbac-proxy-web" pod "7e0ecffe-75b6-4a90-a4df-baa32433ce85" (UID: "7e0ecffe-75b6-4a90-a4df-baa32433ce85"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:48:42.348184 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.348160 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-kube-rbac-proxy" pod "7e0ecffe-75b6-4a90-a4df-baa32433ce85" (UID: "7e0ecffe-75b6-4a90-a4df-baa32433ce85"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:48:42.348773 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.348750 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0ecffe-75b6-4a90-a4df-baa32433ce85-kube-api-access-42tg7" pod "7e0ecffe-75b6-4a90-a4df-baa32433ce85" (UID: "7e0ecffe-75b6-4a90-a4df-baa32433ce85"). InnerVolumeSpecName "kube-api-access-42tg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:48:42.349042 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.349019 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-metrics-client-certs" pod "7e0ecffe-75b6-4a90-a4df-baa32433ce85" (UID: "7e0ecffe-75b6-4a90-a4df-baa32433ce85"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:48:42.356804 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.356780 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-web-config" pod "7e0ecffe-75b6-4a90-a4df-baa32433ce85" (UID: "7e0ecffe-75b6-4a90-a4df-baa32433ce85"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:48:42.445928 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.445896 4581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:42.445928 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.445924 4581 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-metrics-client-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:42.446137 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.445939 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42tg7\" (UniqueName: \"kubernetes.io/projected/7e0ecffe-75b6-4a90-a4df-baa32433ce85-kube-api-access-42tg7\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:42.446137 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.445952 4581 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-grpc-tls\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:42.446137 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.445964 4581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:42.446137 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.445978 4581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:42.446137 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.446010 4581 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:42.446137 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.446023 4581 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-config\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:42.446137 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.446037 4581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7e0ecffe-75b6-4a90-a4df-baa32433ce85-tls-assets\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:42.446137 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.446057 4581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-web-config\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:42.446137 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.446069 4581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e0ecffe-75b6-4a90-a4df-baa32433ce85-config-out\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:42.446137 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.446081 4581 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e0ecffe-75b6-4a90-a4df-baa32433ce85-secret-kube-rbac-proxy\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:42.446137 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.446094 4581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7e0ecffe-75b6-4a90-a4df-baa32433ce85-prometheus-k8s-db\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:42.627478 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.627446 4581 generic.go:356] "Generic (PLEG): container finished" podID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerID="1bbf40bb640e17fd2b731d23ce3d840544587bd8c988387e8147e185a6cd6a6e" exitCode=0 Nov 04 04:48:42.627664 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.627507 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7e0ecffe-75b6-4a90-a4df-baa32433ce85","Type":"ContainerDied","Data":"1bbf40bb640e17fd2b731d23ce3d840544587bd8c988387e8147e185a6cd6a6e"} Nov 04 04:48:42.627664 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.627565 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7e0ecffe-75b6-4a90-a4df-baa32433ce85","Type":"ContainerDied","Data":"13643c64fd2ba8ecd79e5305f724c1d18631a64607f66802ca2be4b960064790"} Nov 04 04:48:42.627664 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.627598 4581 scope.go:122] "RemoveContainer" containerID="22cf1454068603c49f99012546e2657ecdf3686e6e538ccdb765999fb9414538" Nov 04 04:48:42.627664 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.627616 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:42.635306 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.635286 4581 scope.go:122] "RemoveContainer" containerID="5cbda64fca81e81b23b68f589fd84fac6bb1362093a2fe75b38ec6584489603d" Nov 04 04:48:42.642067 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.642039 4581 scope.go:122] "RemoveContainer" containerID="1bbf40bb640e17fd2b731d23ce3d840544587bd8c988387e8147e185a6cd6a6e" Nov 04 04:48:42.648545 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.648530 4581 scope.go:122] "RemoveContainer" containerID="c9ef1308474db7101232fe717122ea21bdb884c36a0fea5ff4f8b50472d8633e" Nov 04 04:48:42.654939 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.654925 4581 scope.go:122] "RemoveContainer" containerID="397189c442d42f76718561db969fe7e47201d797d4b11da43132620a93503948" Nov 04 04:48:42.661321 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.661307 4581 scope.go:122] "RemoveContainer" containerID="eae0651623d491be0c19c9324f78475518e6da10aa58ba0c64d34601967e45d6" Nov 04 04:48:42.668249 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.668235 4581 scope.go:122] "RemoveContainer" containerID="4cf25f2a6bc820a1aa7ac92d7fdd9514fed80dadbd670eddb97a9f36b87534c4" Nov 04 04:48:42.675078 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.675060 4581 scope.go:122] "RemoveContainer" containerID="22cf1454068603c49f99012546e2657ecdf3686e6e538ccdb765999fb9414538" Nov 04 04:48:42.675369 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:48:42.675352 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22cf1454068603c49f99012546e2657ecdf3686e6e538ccdb765999fb9414538\": container with ID starting with 22cf1454068603c49f99012546e2657ecdf3686e6e538ccdb765999fb9414538 not found: ID does not exist" containerID="22cf1454068603c49f99012546e2657ecdf3686e6e538ccdb765999fb9414538" Nov 04 04:48:42.675411 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.675379 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22cf1454068603c49f99012546e2657ecdf3686e6e538ccdb765999fb9414538"} err="failed to get container status \"22cf1454068603c49f99012546e2657ecdf3686e6e538ccdb765999fb9414538\": rpc error: code = NotFound desc = could not find container \"22cf1454068603c49f99012546e2657ecdf3686e6e538ccdb765999fb9414538\": container with ID starting with 22cf1454068603c49f99012546e2657ecdf3686e6e538ccdb765999fb9414538 not found: ID does not exist" Nov 04 04:48:42.675411 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.675399 4581 scope.go:122] "RemoveContainer" containerID="5cbda64fca81e81b23b68f589fd84fac6bb1362093a2fe75b38ec6584489603d" Nov 04 04:48:42.675631 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:48:42.675616 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cbda64fca81e81b23b68f589fd84fac6bb1362093a2fe75b38ec6584489603d\": container with ID starting with 5cbda64fca81e81b23b68f589fd84fac6bb1362093a2fe75b38ec6584489603d not found: ID does not exist" containerID="5cbda64fca81e81b23b68f589fd84fac6bb1362093a2fe75b38ec6584489603d" Nov 04 04:48:42.675677 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.675637 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cbda64fca81e81b23b68f589fd84fac6bb1362093a2fe75b38ec6584489603d"} err="failed to get container status \"5cbda64fca81e81b23b68f589fd84fac6bb1362093a2fe75b38ec6584489603d\": rpc error: code = NotFound desc = could not find container \"5cbda64fca81e81b23b68f589fd84fac6bb1362093a2fe75b38ec6584489603d\": container with ID starting with 5cbda64fca81e81b23b68f589fd84fac6bb1362093a2fe75b38ec6584489603d not found: ID does not exist" Nov 04 04:48:42.675677 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.675654 4581 scope.go:122] "RemoveContainer" containerID="1bbf40bb640e17fd2b731d23ce3d840544587bd8c988387e8147e185a6cd6a6e" Nov 04 04:48:42.675912 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:48:42.675896 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bbf40bb640e17fd2b731d23ce3d840544587bd8c988387e8147e185a6cd6a6e\": container with ID starting with 1bbf40bb640e17fd2b731d23ce3d840544587bd8c988387e8147e185a6cd6a6e not found: ID does not exist" containerID="1bbf40bb640e17fd2b731d23ce3d840544587bd8c988387e8147e185a6cd6a6e" Nov 04 04:48:42.675952 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.675917 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bbf40bb640e17fd2b731d23ce3d840544587bd8c988387e8147e185a6cd6a6e"} err="failed to get container status \"1bbf40bb640e17fd2b731d23ce3d840544587bd8c988387e8147e185a6cd6a6e\": rpc error: code = NotFound desc = could not find container \"1bbf40bb640e17fd2b731d23ce3d840544587bd8c988387e8147e185a6cd6a6e\": container with ID starting with 1bbf40bb640e17fd2b731d23ce3d840544587bd8c988387e8147e185a6cd6a6e not found: ID does not exist" Nov 04 04:48:42.675952 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.675931 4581 scope.go:122] "RemoveContainer" containerID="c9ef1308474db7101232fe717122ea21bdb884c36a0fea5ff4f8b50472d8633e" Nov 04 04:48:42.676183 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:48:42.676164 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9ef1308474db7101232fe717122ea21bdb884c36a0fea5ff4f8b50472d8633e\": container with ID starting with c9ef1308474db7101232fe717122ea21bdb884c36a0fea5ff4f8b50472d8633e not found: ID does not exist" containerID="c9ef1308474db7101232fe717122ea21bdb884c36a0fea5ff4f8b50472d8633e" Nov 04 04:48:42.676233 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.676190 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9ef1308474db7101232fe717122ea21bdb884c36a0fea5ff4f8b50472d8633e"} err="failed to get container status \"c9ef1308474db7101232fe717122ea21bdb884c36a0fea5ff4f8b50472d8633e\": rpc error: code = NotFound desc = could not find container \"c9ef1308474db7101232fe717122ea21bdb884c36a0fea5ff4f8b50472d8633e\": container with ID starting with c9ef1308474db7101232fe717122ea21bdb884c36a0fea5ff4f8b50472d8633e not found: ID does not exist" Nov 04 04:48:42.676233 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.676203 4581 scope.go:122] "RemoveContainer" containerID="397189c442d42f76718561db969fe7e47201d797d4b11da43132620a93503948" Nov 04 04:48:42.676452 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:48:42.676435 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"397189c442d42f76718561db969fe7e47201d797d4b11da43132620a93503948\": container with ID starting with 397189c442d42f76718561db969fe7e47201d797d4b11da43132620a93503948 not found: ID does not exist" containerID="397189c442d42f76718561db969fe7e47201d797d4b11da43132620a93503948" Nov 04 04:48:42.676508 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.676456 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"397189c442d42f76718561db969fe7e47201d797d4b11da43132620a93503948"} err="failed to get container status \"397189c442d42f76718561db969fe7e47201d797d4b11da43132620a93503948\": rpc error: code = NotFound desc = could not find container \"397189c442d42f76718561db969fe7e47201d797d4b11da43132620a93503948\": container with ID starting with 397189c442d42f76718561db969fe7e47201d797d4b11da43132620a93503948 not found: ID does not exist" Nov 04 04:48:42.676508 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.676470 4581 scope.go:122] "RemoveContainer" containerID="eae0651623d491be0c19c9324f78475518e6da10aa58ba0c64d34601967e45d6" Nov 04 04:48:42.676718 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:48:42.676700 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eae0651623d491be0c19c9324f78475518e6da10aa58ba0c64d34601967e45d6\": container with ID starting with eae0651623d491be0c19c9324f78475518e6da10aa58ba0c64d34601967e45d6 not found: ID does not exist" containerID="eae0651623d491be0c19c9324f78475518e6da10aa58ba0c64d34601967e45d6" Nov 04 04:48:42.676784 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.676727 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eae0651623d491be0c19c9324f78475518e6da10aa58ba0c64d34601967e45d6"} err="failed to get container status \"eae0651623d491be0c19c9324f78475518e6da10aa58ba0c64d34601967e45d6\": rpc error: code = NotFound desc = could not find container \"eae0651623d491be0c19c9324f78475518e6da10aa58ba0c64d34601967e45d6\": container with ID starting with eae0651623d491be0c19c9324f78475518e6da10aa58ba0c64d34601967e45d6 not found: ID does not exist" Nov 04 04:48:42.676784 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.676750 4581 scope.go:122] "RemoveContainer" containerID="4cf25f2a6bc820a1aa7ac92d7fdd9514fed80dadbd670eddb97a9f36b87534c4" Nov 04 04:48:42.677054 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:48:42.676974 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf25f2a6bc820a1aa7ac92d7fdd9514fed80dadbd670eddb97a9f36b87534c4\": container with ID starting with 4cf25f2a6bc820a1aa7ac92d7fdd9514fed80dadbd670eddb97a9f36b87534c4 not found: ID does not exist" containerID="4cf25f2a6bc820a1aa7ac92d7fdd9514fed80dadbd670eddb97a9f36b87534c4" Nov 04 04:48:42.677111 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.677068 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf25f2a6bc820a1aa7ac92d7fdd9514fed80dadbd670eddb97a9f36b87534c4"} err="failed to get container status \"4cf25f2a6bc820a1aa7ac92d7fdd9514fed80dadbd670eddb97a9f36b87534c4\": rpc error: code = NotFound desc = could not find container \"4cf25f2a6bc820a1aa7ac92d7fdd9514fed80dadbd670eddb97a9f36b87534c4\": container with ID starting with 4cf25f2a6bc820a1aa7ac92d7fdd9514fed80dadbd670eddb97a9f36b87534c4 not found: ID does not exist" Nov 04 04:48:42.798173 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.798141 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 04:48:42.804184 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.804160 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 04:48:42.849082 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.849049 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 04:48:42.849355 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.849342 4581 cpu_manager.go:414] "containerMap: removing container" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="thanos-sidecar" Nov 04 04:48:42.849392 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.849358 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="thanos-sidecar" Nov 04 04:48:42.849392 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.849366 4581 cpu_manager.go:414] "containerMap: removing container" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="kube-rbac-proxy" Nov 04 04:48:42.849392 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.849370 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="kube-rbac-proxy" Nov 04 04:48:42.849392 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.849380 4581 cpu_manager.go:414] "containerMap: removing container" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="kube-rbac-proxy-web" Nov 04 04:48:42.849392 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.849385 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="kube-rbac-proxy-web" Nov 04 04:48:42.849539 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.849400 4581 cpu_manager.go:414] "containerMap: removing container" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="init-config-reloader" Nov 04 04:48:42.849539 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.849405 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="init-config-reloader" Nov 04 04:48:42.849539 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.849421 4581 cpu_manager.go:414] "containerMap: removing container" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="kube-rbac-proxy-thanos" Nov 04 04:48:42.849539 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.849426 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="kube-rbac-proxy-thanos" Nov 04 04:48:42.849539 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.849438 4581 cpu_manager.go:414] "containerMap: removing container" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="prometheus" Nov 04 04:48:42.849539 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.849442 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="prometheus" Nov 04 04:48:42.849539 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.849453 4581 cpu_manager.go:414] "containerMap: removing container" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="config-reloader" Nov 04 04:48:42.849539 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.849457 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="config-reloader" Nov 04 04:48:42.849539 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.849499 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="kube-rbac-proxy-thanos" Nov 04 04:48:42.849539 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.849510 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="prometheus" Nov 04 04:48:42.849539 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.849517 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="config-reloader" Nov 04 04:48:42.849539 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.849522 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="thanos-sidecar" Nov 04 04:48:42.849539 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.849527 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="kube-rbac-proxy" Nov 04 04:48:42.849539 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.849532 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" containerName="kube-rbac-proxy-web" Nov 04 04:48:42.853068 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.853049 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:42.906348 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.906275 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 04 04:48:42.906473 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.906377 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 04 04:48:42.906473 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.906393 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 04 04:48:42.906577 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.906500 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 04 04:48:42.906644 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.906632 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 04 04:48:42.906694 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.906664 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 04 04:48:42.906752 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.906696 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 04 04:48:42.906752 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.906716 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 04 04:48:42.906853 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.906759 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 04 04:48:42.906853 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.906739 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 04 04:48:42.906853 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.906768 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7h8bv\"" Nov 04 04:48:42.906951 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.906897 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 04 04:48:42.906951 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.906936 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 04 04:48:42.907084 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.906971 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-pdrn1nm48qj1\"" Nov 04 04:48:42.949709 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.949684 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97f1373d-ba83-44f9-bb25-31ada871c58b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:42.949812 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.949711 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/97f1373d-ba83-44f9-bb25-31ada871c58b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:42.949812 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.949732 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:42.949812 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.949781 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:42.949951 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.949816 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-web-config\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:42.949951 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.949872 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97f1373d-ba83-44f9-bb25-31ada871c58b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:42.949951 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.949916 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/97f1373d-ba83-44f9-bb25-31ada871c58b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:42.950115 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.949982 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:42.950115 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.950046 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:42.950115 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.950080 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/97f1373d-ba83-44f9-bb25-31ada871c58b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:42.950211 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.950190 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:42.950243 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.950213 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/97f1373d-ba83-44f9-bb25-31ada871c58b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:42.950272 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.950237 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97f1373d-ba83-44f9-bb25-31ada871c58b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:42.950304 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.950279 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:42.950347 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.950329 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:42.950382 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.950357 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-config\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:42.950382 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.950372 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsnl8\" (UniqueName: \"kubernetes.io/projected/97f1373d-ba83-44f9-bb25-31ada871c58b-kube-api-access-xsnl8\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:42.950446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.950390 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97f1373d-ba83-44f9-bb25-31ada871c58b-config-out\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:42.950446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.950431 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97f1373d-ba83-44f9-bb25-31ada871c58b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:42.950517 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.950447 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97f1373d-ba83-44f9-bb25-31ada871c58b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:42.961341 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.961314 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 04:48:42.961977 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.961947 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 04 04:48:42.964134 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:42.964036 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 04 04:48:43.051467 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.051436 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.051620 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.051473 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/97f1373d-ba83-44f9-bb25-31ada871c58b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.051620 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.051506 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97f1373d-ba83-44f9-bb25-31ada871c58b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.051620 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.051528 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.051620 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.051553 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.051620 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.051577 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-config\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.051620 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.051598 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsnl8\" (UniqueName: \"kubernetes.io/projected/97f1373d-ba83-44f9-bb25-31ada871c58b-kube-api-access-xsnl8\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.051919 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.051720 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97f1373d-ba83-44f9-bb25-31ada871c58b-config-out\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.051919 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.051765 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97f1373d-ba83-44f9-bb25-31ada871c58b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.051919 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.051791 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97f1373d-ba83-44f9-bb25-31ada871c58b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.051919 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.051823 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97f1373d-ba83-44f9-bb25-31ada871c58b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.051919 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.051846 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/97f1373d-ba83-44f9-bb25-31ada871c58b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.051919 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.051875 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.051919 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.051902 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.052298 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.051929 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-web-config\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.052298 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.051961 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97f1373d-ba83-44f9-bb25-31ada871c58b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.052298 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.052025 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/97f1373d-ba83-44f9-bb25-31ada871c58b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.052298 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.052062 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.052298 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.052098 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.052298 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.052133 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/97f1373d-ba83-44f9-bb25-31ada871c58b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.052298 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.052219 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/97f1373d-ba83-44f9-bb25-31ada871c58b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.052628 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.052550 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97f1373d-ba83-44f9-bb25-31ada871c58b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.052628 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.052600 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97f1373d-ba83-44f9-bb25-31ada871c58b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.052899 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.052872 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/97f1373d-ba83-44f9-bb25-31ada871c58b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.054102 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.054074 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/97f1373d-ba83-44f9-bb25-31ada871c58b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.055275 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.054704 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.055275 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.055021 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97f1373d-ba83-44f9-bb25-31ada871c58b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.055275 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.055203 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97f1373d-ba83-44f9-bb25-31ada871c58b-config-out\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.055275 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.055234 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.055629 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.055587 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.055924 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.055903 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-config\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.056158 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.056136 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.056242 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.056228 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.056852 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.056835 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.057350 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.057328 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97f1373d-ba83-44f9-bb25-31ada871c58b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.057788 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.057763 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.058022 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.058000 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/97f1373d-ba83-44f9-bb25-31ada871c58b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.058099 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.057976 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97f1373d-ba83-44f9-bb25-31ada871c58b-web-config\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.058659 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.058643 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97f1373d-ba83-44f9-bb25-31ada871c58b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.108796 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.108769 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsnl8\" (UniqueName: \"kubernetes.io/projected/97f1373d-ba83-44f9-bb25-31ada871c58b-kube-api-access-xsnl8\") pod \"prometheus-k8s-0\" (UID: \"97f1373d-ba83-44f9-bb25-31ada871c58b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.170099 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.170026 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.229665 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.229582 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e0ecffe-75b6-4a90-a4df-baa32433ce85" path="/var/lib/kubelet/pods/7e0ecffe-75b6-4a90-a4df-baa32433ce85/volumes" Nov 04 04:48:43.597616 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.597594 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 04:48:43.600100 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:48:43.600066 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f1373d_ba83_44f9_bb25_31ada871c58b.slice/crio-dfb7f4c14f6e9b843baea9550a7b5d882e208676aebd7e43d7556aa74e29c35c WatchSource:0}: Error finding container dfb7f4c14f6e9b843baea9550a7b5d882e208676aebd7e43d7556aa74e29c35c: Status 404 returned error can't find the container with id dfb7f4c14f6e9b843baea9550a7b5d882e208676aebd7e43d7556aa74e29c35c Nov 04 04:48:43.632375 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:43.632350 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97f1373d-ba83-44f9-bb25-31ada871c58b","Type":"ContainerStarted","Data":"dfb7f4c14f6e9b843baea9550a7b5d882e208676aebd7e43d7556aa74e29c35c"} Nov 04 04:48:44.637793 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:44.637763 4581 generic.go:356] "Generic (PLEG): container finished" podID="97f1373d-ba83-44f9-bb25-31ada871c58b" containerID="bbbe428ea039af7ddd8a304d6eaac983b2e5fb563a90c3beb0c7ac76082e0ad0" exitCode=0 Nov 04 04:48:44.638217 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:44.637801 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97f1373d-ba83-44f9-bb25-31ada871c58b","Type":"ContainerDied","Data":"bbbe428ea039af7ddd8a304d6eaac983b2e5fb563a90c3beb0c7ac76082e0ad0"} Nov 04 04:48:45.645263 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:45.645230 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97f1373d-ba83-44f9-bb25-31ada871c58b","Type":"ContainerStarted","Data":"1962a55c1f00982cc61b724915947e1bf6a04293fe56d1d39777505bb1b590dd"} Nov 04 04:48:45.645635 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:45.645294 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97f1373d-ba83-44f9-bb25-31ada871c58b","Type":"ContainerStarted","Data":"8eb8339fa54c57e0378c7aa67255bdc60bd0647246b1b2a090798aa9b3aadbbe"} Nov 04 04:48:45.645635 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:45.645319 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97f1373d-ba83-44f9-bb25-31ada871c58b","Type":"ContainerStarted","Data":"8e84447a37fa6098a8f0573fd2f33d430305c0b24040f997b23f4d9ab13d0f4b"} Nov 04 04:48:45.645635 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:45.645342 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97f1373d-ba83-44f9-bb25-31ada871c58b","Type":"ContainerStarted","Data":"7862a2ff89f11fd6b27b68b73d6969a87ec9959e78581d7d49177629fe5aaa68"} Nov 04 04:48:45.645635 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:45.645371 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97f1373d-ba83-44f9-bb25-31ada871c58b","Type":"ContainerStarted","Data":"7767a870cce3ec82265acdf93f0d81c8f38555dfec49d724cdb523f5480af94f"} Nov 04 04:48:45.645635 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:45.645402 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97f1373d-ba83-44f9-bb25-31ada871c58b","Type":"ContainerStarted","Data":"3fd76dd01b6bf2199f6b82a85aebfcb682de294028c7027482f99c0a88f5cb61"} Nov 04 04:48:45.772355 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:45.772296 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.756409711 podStartE2EDuration="3.772282609s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.872898ms" isStatefulPod=false podCreationTimestamp="2026-11-04 04:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:48:45.770661531 +0000 UTC m=+84.661518582" watchObservedRunningTime="2026-11-04 04:48:45.772282609 +0000 UTC m=+84.663139629" Nov 04 04:48:47.885075 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:48:47.885040 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 04:48:48.170416 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:48.170345 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:56.663762 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:48:56.663724 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:49:17.898173 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:49:17.898137 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 04:49:43.170923 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:49:43.170830 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:49:43.186921 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:49:43.186892 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:49:43.873233 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:49:43.873204 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:49:47.886240 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:49:47.886199 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 04:49:56.999354 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:49:56.999309 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:50:08.152596 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:08.152540 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:50:08.330771 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:08.330730 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:50:17.885193 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:50:17.885153 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 04:50:37.970516 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:37.970473 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:50:38.146114 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:38.146076 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:50:47.884724 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:50:47.884685 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 04:50:51.327551 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:51.327501 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:50:51.537245 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:51.537210 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:50:52.670467 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:52.670435 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:50:53.808360 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:53.808323 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:50:57.339203 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:57.339175 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:50:58.951974 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:58.951940 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s"] Nov 04 04:50:58.955372 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:58.955351 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s" Nov 04 04:50:59.061719 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:59.061688 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-smwxx\"" Nov 04 04:50:59.061891 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:59.061737 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 04 04:50:59.061891 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:59.061774 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 04 04:50:59.062800 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:59.062779 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s"] Nov 04 04:50:59.069850 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:59.069834 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b70ed28-93df-4bfc-9c1d-e992afc5bfc1-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s\" (UID: \"1b70ed28-93df-4bfc-9c1d-e992afc5bfc1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s" Nov 04 04:50:59.069907 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:59.069867 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b70ed28-93df-4bfc-9c1d-e992afc5bfc1-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s\" (UID: \"1b70ed28-93df-4bfc-9c1d-e992afc5bfc1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s" Nov 04 04:50:59.069975 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:59.069956 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg928\" (UniqueName: \"kubernetes.io/projected/1b70ed28-93df-4bfc-9c1d-e992afc5bfc1-kube-api-access-rg928\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s\" (UID: \"1b70ed28-93df-4bfc-9c1d-e992afc5bfc1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s" Nov 04 04:50:59.125895 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:59.125864 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:50:59.170797 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:59.170764 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg928\" (UniqueName: \"kubernetes.io/projected/1b70ed28-93df-4bfc-9c1d-e992afc5bfc1-kube-api-access-rg928\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s\" (UID: \"1b70ed28-93df-4bfc-9c1d-e992afc5bfc1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s" Nov 04 04:50:59.170945 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:59.170833 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b70ed28-93df-4bfc-9c1d-e992afc5bfc1-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s\" (UID: \"1b70ed28-93df-4bfc-9c1d-e992afc5bfc1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s" Nov 04 04:50:59.170945 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:59.170873 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b70ed28-93df-4bfc-9c1d-e992afc5bfc1-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s\" (UID: \"1b70ed28-93df-4bfc-9c1d-e992afc5bfc1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s" Nov 04 04:50:59.171248 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:59.171228 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b70ed28-93df-4bfc-9c1d-e992afc5bfc1-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s\" (UID: \"1b70ed28-93df-4bfc-9c1d-e992afc5bfc1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s" Nov 04 04:50:59.171317 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:59.171261 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b70ed28-93df-4bfc-9c1d-e992afc5bfc1-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s\" (UID: \"1b70ed28-93df-4bfc-9c1d-e992afc5bfc1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s" Nov 04 04:50:59.228887 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:59.228828 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg928\" (UniqueName: \"kubernetes.io/projected/1b70ed28-93df-4bfc-9c1d-e992afc5bfc1-kube-api-access-rg928\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s\" (UID: \"1b70ed28-93df-4bfc-9c1d-e992afc5bfc1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s" Nov 04 04:50:59.261137 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:59.261112 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:50:59.269948 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:59.269928 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s" Nov 04 04:50:59.639180 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:50:59.639069 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s"] Nov 04 04:50:59.641901 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:50:59.641875 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b70ed28_93df_4bfc_9c1d_e992afc5bfc1.slice/crio-734b38cba0e1dcf1f27e2e0aa7021944e6bbb2bab2fa7eabe89d92c2bfb2cbe5 WatchSource:0}: Error finding container 734b38cba0e1dcf1f27e2e0aa7021944e6bbb2bab2fa7eabe89d92c2bfb2cbe5: Status 404 returned error can't find the container with id 734b38cba0e1dcf1f27e2e0aa7021944e6bbb2bab2fa7eabe89d92c2bfb2cbe5 Nov 04 04:51:00.091417 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:00.091338 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:00.127889 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:00.127857 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s" event={"ID":"1b70ed28-93df-4bfc-9c1d-e992afc5bfc1","Type":"ContainerStarted","Data":"734b38cba0e1dcf1f27e2e0aa7021944e6bbb2bab2fa7eabe89d92c2bfb2cbe5"} Nov 04 04:51:00.395116 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:00.395076 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:00.524538 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:00.524499 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:00.648089 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:00.648007 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:00.861846 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:00.861812 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t"] Nov 04 04:51:00.865377 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:00.865353 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t" Nov 04 04:51:00.922503 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:00.920874 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c"] Nov 04 04:51:00.924789 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:00.924724 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c" Nov 04 04:51:00.970153 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:00.970111 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-cfxkm\"" Nov 04 04:51:00.971523 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:00.971497 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t"] Nov 04 04:51:00.974070 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:00.974045 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn"] Nov 04 04:51:00.977578 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:00.977557 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn" Nov 04 04:51:00.984652 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:00.984384 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bztmj\" (UniqueName: \"kubernetes.io/projected/30217920-93bc-4930-92d2-b2e064730a3d-kube-api-access-bztmj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t\" (UID: \"30217920-93bc-4930-92d2-b2e064730a3d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t" Nov 04 04:51:00.984652 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:00.984439 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30217920-93bc-4930-92d2-b2e064730a3d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t\" (UID: \"30217920-93bc-4930-92d2-b2e064730a3d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t" Nov 04 04:51:00.984652 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:00.984528 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30217920-93bc-4930-92d2-b2e064730a3d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t\" (UID: \"30217920-93bc-4930-92d2-b2e064730a3d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t" Nov 04 04:51:00.987380 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:00.987361 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c"] Nov 04 04:51:01.021751 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.021719 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn"] Nov 04 04:51:01.023221 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.023190 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-p98qn\"" Nov 04 04:51:01.031164 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.031140 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-c5wvt\"" Nov 04 04:51:01.085084 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.085046 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06f04549-d6d9-4916-91b2-c8e97c52e716-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn\" (UID: \"06f04549-d6d9-4916-91b2-c8e97c52e716\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn" Nov 04 04:51:01.085363 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.085100 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8glqg\" (UniqueName: \"kubernetes.io/projected/06f04549-d6d9-4916-91b2-c8e97c52e716-kube-api-access-8glqg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn\" (UID: \"06f04549-d6d9-4916-91b2-c8e97c52e716\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn" Nov 04 04:51:01.085363 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.085212 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bztmj\" (UniqueName: \"kubernetes.io/projected/30217920-93bc-4930-92d2-b2e064730a3d-kube-api-access-bztmj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t\" (UID: \"30217920-93bc-4930-92d2-b2e064730a3d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t" Nov 04 04:51:01.085363 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.085256 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fa61466-9f70-403a-ab95-149e7a9a30de-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c\" (UID: \"2fa61466-9f70-403a-ab95-149e7a9a30de\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c" Nov 04 04:51:01.085363 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.085310 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30217920-93bc-4930-92d2-b2e064730a3d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t\" (UID: \"30217920-93bc-4930-92d2-b2e064730a3d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t" Nov 04 04:51:01.085577 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.085477 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s62t2\" (UniqueName: \"kubernetes.io/projected/2fa61466-9f70-403a-ab95-149e7a9a30de-kube-api-access-s62t2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c\" (UID: \"2fa61466-9f70-403a-ab95-149e7a9a30de\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c" Nov 04 04:51:01.085644 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.085599 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06f04549-d6d9-4916-91b2-c8e97c52e716-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn\" (UID: \"06f04549-d6d9-4916-91b2-c8e97c52e716\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn" Nov 04 04:51:01.085644 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.085628 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fa61466-9f70-403a-ab95-149e7a9a30de-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c\" (UID: \"2fa61466-9f70-403a-ab95-149e7a9a30de\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c" Nov 04 04:51:01.085803 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.085708 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30217920-93bc-4930-92d2-b2e064730a3d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t\" (UID: \"30217920-93bc-4930-92d2-b2e064730a3d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t" Nov 04 04:51:01.085803 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.085743 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30217920-93bc-4930-92d2-b2e064730a3d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t\" (UID: \"30217920-93bc-4930-92d2-b2e064730a3d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t" Nov 04 04:51:01.086021 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.086005 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30217920-93bc-4930-92d2-b2e064730a3d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t\" (UID: \"30217920-93bc-4930-92d2-b2e064730a3d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t" Nov 04 04:51:01.143742 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.143694 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bztmj\" (UniqueName: \"kubernetes.io/projected/30217920-93bc-4930-92d2-b2e064730a3d-kube-api-access-bztmj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t\" (UID: \"30217920-93bc-4930-92d2-b2e064730a3d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t" Nov 04 04:51:01.183637 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.183558 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t" Nov 04 04:51:01.186521 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.186496 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s62t2\" (UniqueName: \"kubernetes.io/projected/2fa61466-9f70-403a-ab95-149e7a9a30de-kube-api-access-s62t2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c\" (UID: \"2fa61466-9f70-403a-ab95-149e7a9a30de\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c" Nov 04 04:51:01.186688 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.186568 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06f04549-d6d9-4916-91b2-c8e97c52e716-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn\" (UID: \"06f04549-d6d9-4916-91b2-c8e97c52e716\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn" Nov 04 04:51:01.186688 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.186591 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fa61466-9f70-403a-ab95-149e7a9a30de-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c\" (UID: \"2fa61466-9f70-403a-ab95-149e7a9a30de\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c" Nov 04 04:51:01.186688 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.186645 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06f04549-d6d9-4916-91b2-c8e97c52e716-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn\" (UID: \"06f04549-d6d9-4916-91b2-c8e97c52e716\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn" Nov 04 04:51:01.186688 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.186673 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8glqg\" (UniqueName: \"kubernetes.io/projected/06f04549-d6d9-4916-91b2-c8e97c52e716-kube-api-access-8glqg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn\" (UID: \"06f04549-d6d9-4916-91b2-c8e97c52e716\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn" Nov 04 04:51:01.186891 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.186710 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fa61466-9f70-403a-ab95-149e7a9a30de-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c\" (UID: \"2fa61466-9f70-403a-ab95-149e7a9a30de\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c" Nov 04 04:51:01.187359 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.187058 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06f04549-d6d9-4916-91b2-c8e97c52e716-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn\" (UID: \"06f04549-d6d9-4916-91b2-c8e97c52e716\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn" Nov 04 04:51:01.187359 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.187106 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06f04549-d6d9-4916-91b2-c8e97c52e716-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn\" (UID: \"06f04549-d6d9-4916-91b2-c8e97c52e716\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn" Nov 04 04:51:01.187359 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.187129 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fa61466-9f70-403a-ab95-149e7a9a30de-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c\" (UID: \"2fa61466-9f70-403a-ab95-149e7a9a30de\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c" Nov 04 04:51:01.187359 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.187100 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fa61466-9f70-403a-ab95-149e7a9a30de-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c\" (UID: \"2fa61466-9f70-403a-ab95-149e7a9a30de\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c" Nov 04 04:51:01.245681 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.245547 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8glqg\" (UniqueName: \"kubernetes.io/projected/06f04549-d6d9-4916-91b2-c8e97c52e716-kube-api-access-8glqg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn\" (UID: \"06f04549-d6d9-4916-91b2-c8e97c52e716\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn" Nov 04 04:51:01.252235 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.251679 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s62t2\" (UniqueName: \"kubernetes.io/projected/2fa61466-9f70-403a-ab95-149e7a9a30de-kube-api-access-s62t2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c\" (UID: \"2fa61466-9f70-403a-ab95-149e7a9a30de\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c" Nov 04 04:51:01.295416 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.295378 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn" Nov 04 04:51:01.543869 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.543778 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c" Nov 04 04:51:01.568697 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.568676 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t"] Nov 04 04:51:01.571893 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:51:01.571858 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30217920_93bc_4930_92d2_b2e064730a3d.slice/crio-f994a281e4705306bd83a7c42f6ad5119fb6507d16a0fd741cafebfc702b2630 WatchSource:0}: Error finding container f994a281e4705306bd83a7c42f6ad5119fb6507d16a0fd741cafebfc702b2630: Status 404 returned error can't find the container with id f994a281e4705306bd83a7c42f6ad5119fb6507d16a0fd741cafebfc702b2630 Nov 04 04:51:01.675750 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.675717 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn"] Nov 04 04:51:01.679205 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:51:01.679173 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f04549_d6d9_4916_91b2_c8e97c52e716.slice/crio-13669743cba7884a5f0ee480fd74fe9e53de4be8e409bc539c9b5a9712b43093 WatchSource:0}: Error finding container 13669743cba7884a5f0ee480fd74fe9e53de4be8e409bc539c9b5a9712b43093: Status 404 returned error can't find the container with id 13669743cba7884a5f0ee480fd74fe9e53de4be8e409bc539c9b5a9712b43093 Nov 04 04:51:01.926531 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:01.926486 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c"] Nov 04 04:51:01.929868 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:51:01.929835 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa61466_9f70_403a_ab95_149e7a9a30de.slice/crio-7d656c684d59163e40caa4897be9719171b08120abb1ab58f8c537d1c755f70f WatchSource:0}: Error finding container 7d656c684d59163e40caa4897be9719171b08120abb1ab58f8c537d1c755f70f: Status 404 returned error can't find the container with id 7d656c684d59163e40caa4897be9719171b08120abb1ab58f8c537d1c755f70f Nov 04 04:51:02.137968 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:02.137880 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn" event={"ID":"06f04549-d6d9-4916-91b2-c8e97c52e716","Type":"ContainerStarted","Data":"13669743cba7884a5f0ee480fd74fe9e53de4be8e409bc539c9b5a9712b43093"} Nov 04 04:51:02.139407 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:02.139379 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t" event={"ID":"30217920-93bc-4930-92d2-b2e064730a3d","Type":"ContainerStarted","Data":"f994a281e4705306bd83a7c42f6ad5119fb6507d16a0fd741cafebfc702b2630"} Nov 04 04:51:02.144130 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:02.144099 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c" event={"ID":"2fa61466-9f70-403a-ab95-149e7a9a30de","Type":"ContainerStarted","Data":"7d656c684d59163e40caa4897be9719171b08120abb1ab58f8c537d1c755f70f"} Nov 04 04:51:04.154862 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:04.154810 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:08.181093 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:08.181062 4581 generic.go:356] "Generic (PLEG): container finished" podID="2fa61466-9f70-403a-ab95-149e7a9a30de" containerID="a3c73be6dae66e2c79be14e27edd0778aff1ede24b3ca53d707f7c1a58e3c31d" exitCode=0 Nov 04 04:51:08.181535 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:08.181144 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c" event={"ID":"2fa61466-9f70-403a-ab95-149e7a9a30de","Type":"ContainerDied","Data":"a3c73be6dae66e2c79be14e27edd0778aff1ede24b3ca53d707f7c1a58e3c31d"} Nov 04 04:51:08.184173 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:08.183476 4581 generic.go:356] "Generic (PLEG): container finished" podID="06f04549-d6d9-4916-91b2-c8e97c52e716" containerID="53074e22dc5a42357ccef9fa1190a42258b5d97c0836e725dafacdee82f8f526" exitCode=0 Nov 04 04:51:08.184173 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:08.183563 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn" event={"ID":"06f04549-d6d9-4916-91b2-c8e97c52e716","Type":"ContainerDied","Data":"53074e22dc5a42357ccef9fa1190a42258b5d97c0836e725dafacdee82f8f526"} Nov 04 04:51:08.185709 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:08.185682 4581 generic.go:356] "Generic (PLEG): container finished" podID="1b70ed28-93df-4bfc-9c1d-e992afc5bfc1" containerID="106b3b076ceeaa9d8ff52a6dd803115a52a7b825d1b645348dedb7ae8e6a7594" exitCode=0 Nov 04 04:51:08.185804 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:08.185789 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s" event={"ID":"1b70ed28-93df-4bfc-9c1d-e992afc5bfc1","Type":"ContainerDied","Data":"106b3b076ceeaa9d8ff52a6dd803115a52a7b825d1b645348dedb7ae8e6a7594"} Nov 04 04:51:10.200580 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:10.200477 4581 generic.go:356] "Generic (PLEG): container finished" podID="06f04549-d6d9-4916-91b2-c8e97c52e716" containerID="74abecb22f542e84f09d161b966774cf75038dc7c3323f448db17306743aa135" exitCode=0 Nov 04 04:51:10.200580 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:10.200552 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn" event={"ID":"06f04549-d6d9-4916-91b2-c8e97c52e716","Type":"ContainerDied","Data":"74abecb22f542e84f09d161b966774cf75038dc7c3323f448db17306743aa135"} Nov 04 04:51:10.208647 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:10.203111 4581 generic.go:356] "Generic (PLEG): container finished" podID="30217920-93bc-4930-92d2-b2e064730a3d" containerID="51a612f44194969a977bd8fe958387dbbe0dc310a2b3d1a06bb6db6a7d0bf8e8" exitCode=0 Nov 04 04:51:10.208647 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:10.203139 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t" event={"ID":"30217920-93bc-4930-92d2-b2e064730a3d","Type":"ContainerDied","Data":"51a612f44194969a977bd8fe958387dbbe0dc310a2b3d1a06bb6db6a7d0bf8e8"} Nov 04 04:51:11.210620 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:11.210585 4581 generic.go:356] "Generic (PLEG): container finished" podID="2fa61466-9f70-403a-ab95-149e7a9a30de" containerID="966fe4eb7797fcaaf5ec56c39368f4cc960f329ae22c4347d45193ddbf2803bd" exitCode=0 Nov 04 04:51:11.211006 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:11.210660 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c" event={"ID":"2fa61466-9f70-403a-ab95-149e7a9a30de","Type":"ContainerDied","Data":"966fe4eb7797fcaaf5ec56c39368f4cc960f329ae22c4347d45193ddbf2803bd"} Nov 04 04:51:12.218431 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:12.218392 4581 generic.go:356] "Generic (PLEG): container finished" podID="30217920-93bc-4930-92d2-b2e064730a3d" containerID="44ba8ca7b29811a4e61770496a9c11303a6b695edce8fe3991dfa0034edec913" exitCode=0 Nov 04 04:51:12.219026 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:12.218500 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t" event={"ID":"30217920-93bc-4930-92d2-b2e064730a3d","Type":"ContainerDied","Data":"44ba8ca7b29811a4e61770496a9c11303a6b695edce8fe3991dfa0034edec913"} Nov 04 04:51:12.221875 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:12.221763 4581 generic.go:356] "Generic (PLEG): container finished" podID="1b70ed28-93df-4bfc-9c1d-e992afc5bfc1" containerID="2d9fdd21d22700b0fb5f9c05784ace0e69f593c230c7d29db58b40de91be79af" exitCode=0 Nov 04 04:51:12.221875 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:12.221822 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s" event={"ID":"1b70ed28-93df-4bfc-9c1d-e992afc5bfc1","Type":"ContainerDied","Data":"2d9fdd21d22700b0fb5f9c05784ace0e69f593c230c7d29db58b40de91be79af"} Nov 04 04:51:14.904925 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:14.904878 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:17.668719 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:17.668682 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:17.871820 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:17.871781 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:17.899818 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:51:17.899777 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 04:51:20.272769 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:20.272732 4581 generic.go:356] "Generic (PLEG): container finished" podID="06f04549-d6d9-4916-91b2-c8e97c52e716" containerID="84d152f1244b066630647d24121ce2656a09f976715ae43ad76f182b57555490" exitCode=0 Nov 04 04:51:20.273240 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:20.272812 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn" event={"ID":"06f04549-d6d9-4916-91b2-c8e97c52e716","Type":"ContainerDied","Data":"84d152f1244b066630647d24121ce2656a09f976715ae43ad76f182b57555490"} Nov 04 04:51:21.279944 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:21.279905 4581 generic.go:356] "Generic (PLEG): container finished" podID="2fa61466-9f70-403a-ab95-149e7a9a30de" containerID="b9fbe1f70931c92962d4d7c3bd16a2aedb6a2b8d129461b832906400dc16d05c" exitCode=0 Nov 04 04:51:21.280364 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:21.280334 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c" event={"ID":"2fa61466-9f70-403a-ab95-149e7a9a30de","Type":"ContainerDied","Data":"b9fbe1f70931c92962d4d7c3bd16a2aedb6a2b8d129461b832906400dc16d05c"} Nov 04 04:51:21.418339 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:21.418311 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn" Nov 04 04:51:21.483424 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:21.483385 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f04549-d6d9-4916-91b2-c8e97c52e716-kube-api-access-8glqg\" (UniqueName: \"kubernetes.io/projected/06f04549-d6d9-4916-91b2-c8e97c52e716-kube-api-access-8glqg\") pod \"06f04549-d6d9-4916-91b2-c8e97c52e716\" (UID: \"06f04549-d6d9-4916-91b2-c8e97c52e716\") " Nov 04 04:51:21.483597 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:21.483451 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f04549-d6d9-4916-91b2-c8e97c52e716-bundle\" (UniqueName: \"kubernetes.io/empty-dir/06f04549-d6d9-4916-91b2-c8e97c52e716-bundle\") pod \"06f04549-d6d9-4916-91b2-c8e97c52e716\" (UID: \"06f04549-d6d9-4916-91b2-c8e97c52e716\") " Nov 04 04:51:21.483597 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:21.483523 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f04549-d6d9-4916-91b2-c8e97c52e716-util\" (UniqueName: \"kubernetes.io/empty-dir/06f04549-d6d9-4916-91b2-c8e97c52e716-util\") pod \"06f04549-d6d9-4916-91b2-c8e97c52e716\" (UID: \"06f04549-d6d9-4916-91b2-c8e97c52e716\") " Nov 04 04:51:21.484203 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:21.484163 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f04549-d6d9-4916-91b2-c8e97c52e716-bundle" pod "06f04549-d6d9-4916-91b2-c8e97c52e716" (UID: "06f04549-d6d9-4916-91b2-c8e97c52e716"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:51:21.484330 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:21.484277 4581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06f04549-d6d9-4916-91b2-c8e97c52e716-bundle\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:51:21.490410 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:21.490375 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f04549-d6d9-4916-91b2-c8e97c52e716-util" pod "06f04549-d6d9-4916-91b2-c8e97c52e716" (UID: "06f04549-d6d9-4916-91b2-c8e97c52e716"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:51:21.494919 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:21.494885 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f04549-d6d9-4916-91b2-c8e97c52e716-kube-api-access-8glqg" pod "06f04549-d6d9-4916-91b2-c8e97c52e716" (UID: "06f04549-d6d9-4916-91b2-c8e97c52e716"). InnerVolumeSpecName "kube-api-access-8glqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:51:21.585468 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:21.585426 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8glqg\" (UniqueName: \"kubernetes.io/projected/06f04549-d6d9-4916-91b2-c8e97c52e716-kube-api-access-8glqg\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:51:21.585468 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:21.585468 4581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06f04549-d6d9-4916-91b2-c8e97c52e716-util\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:51:22.285683 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:22.285646 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s" event={"ID":"1b70ed28-93df-4bfc-9c1d-e992afc5bfc1","Type":"ContainerStarted","Data":"c427ef6613fd0c2fb20eb203f11f719bcb06f145c17f926f1f864fca4864347e"} Nov 04 04:51:22.287585 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:22.287555 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn" event={"ID":"06f04549-d6d9-4916-91b2-c8e97c52e716","Type":"ContainerDied","Data":"13669743cba7884a5f0ee480fd74fe9e53de4be8e409bc539c9b5a9712b43093"} Nov 04 04:51:22.287704 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:22.287601 4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13669743cba7884a5f0ee480fd74fe9e53de4be8e409bc539c9b5a9712b43093" Nov 04 04:51:22.287704 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:22.287565 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x8pkn" Nov 04 04:51:22.289813 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:22.289780 4581 generic.go:356] "Generic (PLEG): container finished" podID="30217920-93bc-4930-92d2-b2e064730a3d" containerID="2d062cda7c6a53f1438a55046239707ef81e6684924e3a9f8359905bc2347c14" exitCode=0 Nov 04 04:51:22.289921 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:22.289847 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t" event={"ID":"30217920-93bc-4930-92d2-b2e064730a3d","Type":"ContainerDied","Data":"2d062cda7c6a53f1438a55046239707ef81e6684924e3a9f8359905bc2347c14"} Nov 04 04:51:22.403537 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:22.403492 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s" podStartSLOduration=3.851414666 podStartE2EDuration="24.403476903s" totalImagesPullingTime="20.552062237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:50:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:51:22.401122104 +0000 UTC m=+241.291979125" watchObservedRunningTime="2026-11-04 04:51:22.403476903 +0000 UTC m=+241.294333923" Nov 04 04:51:22.464365 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:22.464309 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c" Nov 04 04:51:22.594376 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:22.594344 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa61466-9f70-403a-ab95-149e7a9a30de-util\" (UniqueName: \"kubernetes.io/empty-dir/2fa61466-9f70-403a-ab95-149e7a9a30de-util\") pod \"2fa61466-9f70-403a-ab95-149e7a9a30de\" (UID: \"2fa61466-9f70-403a-ab95-149e7a9a30de\") " Nov 04 04:51:22.594533 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:22.594457 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa61466-9f70-403a-ab95-149e7a9a30de-kube-api-access-s62t2\" (UniqueName: \"kubernetes.io/projected/2fa61466-9f70-403a-ab95-149e7a9a30de-kube-api-access-s62t2\") pod \"2fa61466-9f70-403a-ab95-149e7a9a30de\" (UID: \"2fa61466-9f70-403a-ab95-149e7a9a30de\") " Nov 04 04:51:22.594533 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:22.594524 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa61466-9f70-403a-ab95-149e7a9a30de-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fa61466-9f70-403a-ab95-149e7a9a30de-bundle\") pod \"2fa61466-9f70-403a-ab95-149e7a9a30de\" (UID: \"2fa61466-9f70-403a-ab95-149e7a9a30de\") " Nov 04 04:51:22.595034 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:22.594978 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa61466-9f70-403a-ab95-149e7a9a30de-bundle" pod "2fa61466-9f70-403a-ab95-149e7a9a30de" (UID: "2fa61466-9f70-403a-ab95-149e7a9a30de"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:51:22.596638 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:22.596606 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa61466-9f70-403a-ab95-149e7a9a30de-kube-api-access-s62t2" pod "2fa61466-9f70-403a-ab95-149e7a9a30de" (UID: "2fa61466-9f70-403a-ab95-149e7a9a30de"). InnerVolumeSpecName "kube-api-access-s62t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:51:22.598598 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:22.598572 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa61466-9f70-403a-ab95-149e7a9a30de-util" pod "2fa61466-9f70-403a-ab95-149e7a9a30de" (UID: "2fa61466-9f70-403a-ab95-149e7a9a30de"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:51:22.695274 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:22.695245 4581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fa61466-9f70-403a-ab95-149e7a9a30de-util\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:51:22.695462 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:22.695273 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s62t2\" (UniqueName: \"kubernetes.io/projected/2fa61466-9f70-403a-ab95-149e7a9a30de-kube-api-access-s62t2\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:51:22.695462 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:22.695297 4581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fa61466-9f70-403a-ab95-149e7a9a30de-bundle\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:51:23.296582 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:23.296547 4581 generic.go:356] "Generic (PLEG): container finished" podID="1b70ed28-93df-4bfc-9c1d-e992afc5bfc1" containerID="c427ef6613fd0c2fb20eb203f11f719bcb06f145c17f926f1f864fca4864347e" exitCode=0 Nov 04 04:51:23.297016 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:23.296626 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s" event={"ID":"1b70ed28-93df-4bfc-9c1d-e992afc5bfc1","Type":"ContainerDied","Data":"c427ef6613fd0c2fb20eb203f11f719bcb06f145c17f926f1f864fca4864347e"} Nov 04 04:51:23.298405 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:23.298385 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c" Nov 04 04:51:23.298405 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:23.298386 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct7k4c" event={"ID":"2fa61466-9f70-403a-ab95-149e7a9a30de","Type":"ContainerDied","Data":"7d656c684d59163e40caa4897be9719171b08120abb1ab58f8c537d1c755f70f"} Nov 04 04:51:23.298607 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:23.298427 4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d656c684d59163e40caa4897be9719171b08120abb1ab58f8c537d1c755f70f" Nov 04 04:51:23.457977 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:23.457949 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t" Nov 04 04:51:23.602072 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:23.602001 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30217920-93bc-4930-92d2-b2e064730a3d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/30217920-93bc-4930-92d2-b2e064730a3d-bundle\") pod \"30217920-93bc-4930-92d2-b2e064730a3d\" (UID: \"30217920-93bc-4930-92d2-b2e064730a3d\") " Nov 04 04:51:23.602208 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:23.602101 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30217920-93bc-4930-92d2-b2e064730a3d-kube-api-access-bztmj\" (UniqueName: \"kubernetes.io/projected/30217920-93bc-4930-92d2-b2e064730a3d-kube-api-access-bztmj\") pod \"30217920-93bc-4930-92d2-b2e064730a3d\" (UID: \"30217920-93bc-4930-92d2-b2e064730a3d\") " Nov 04 04:51:23.602208 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:23.602148 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30217920-93bc-4930-92d2-b2e064730a3d-util\" (UniqueName: \"kubernetes.io/empty-dir/30217920-93bc-4930-92d2-b2e064730a3d-util\") pod \"30217920-93bc-4930-92d2-b2e064730a3d\" (UID: \"30217920-93bc-4930-92d2-b2e064730a3d\") " Nov 04 04:51:23.602572 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:23.602550 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30217920-93bc-4930-92d2-b2e064730a3d-bundle" pod "30217920-93bc-4930-92d2-b2e064730a3d" (UID: "30217920-93bc-4930-92d2-b2e064730a3d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:51:23.604372 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:23.604341 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30217920-93bc-4930-92d2-b2e064730a3d-kube-api-access-bztmj" pod "30217920-93bc-4930-92d2-b2e064730a3d" (UID: "30217920-93bc-4930-92d2-b2e064730a3d"). InnerVolumeSpecName "kube-api-access-bztmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:51:23.607609 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:23.607583 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30217920-93bc-4930-92d2-b2e064730a3d-util" pod "30217920-93bc-4930-92d2-b2e064730a3d" (UID: "30217920-93bc-4930-92d2-b2e064730a3d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:51:23.702888 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:23.702862 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bztmj\" (UniqueName: \"kubernetes.io/projected/30217920-93bc-4930-92d2-b2e064730a3d-kube-api-access-bztmj\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:51:23.702888 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:23.702886 4581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30217920-93bc-4930-92d2-b2e064730a3d-util\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:51:23.703078 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:23.702897 4581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30217920-93bc-4930-92d2-b2e064730a3d-bundle\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:51:24.304262 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:24.304233 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t" Nov 04 04:51:24.304262 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:24.304254 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bc54t" event={"ID":"30217920-93bc-4930-92d2-b2e064730a3d","Type":"ContainerDied","Data":"f994a281e4705306bd83a7c42f6ad5119fb6507d16a0fd741cafebfc702b2630"} Nov 04 04:51:24.304692 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:24.304293 4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f994a281e4705306bd83a7c42f6ad5119fb6507d16a0fd741cafebfc702b2630" Nov 04 04:51:24.428583 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:24.428562 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s" Nov 04 04:51:24.510227 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:24.510197 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b70ed28-93df-4bfc-9c1d-e992afc5bfc1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b70ed28-93df-4bfc-9c1d-e992afc5bfc1-bundle\") pod \"1b70ed28-93df-4bfc-9c1d-e992afc5bfc1\" (UID: \"1b70ed28-93df-4bfc-9c1d-e992afc5bfc1\") " Nov 04 04:51:24.510227 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:24.510230 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b70ed28-93df-4bfc-9c1d-e992afc5bfc1-util\" (UniqueName: \"kubernetes.io/empty-dir/1b70ed28-93df-4bfc-9c1d-e992afc5bfc1-util\") pod \"1b70ed28-93df-4bfc-9c1d-e992afc5bfc1\" (UID: \"1b70ed28-93df-4bfc-9c1d-e992afc5bfc1\") " Nov 04 04:51:24.510423 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:24.510290 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b70ed28-93df-4bfc-9c1d-e992afc5bfc1-kube-api-access-rg928\" (UniqueName: \"kubernetes.io/projected/1b70ed28-93df-4bfc-9c1d-e992afc5bfc1-kube-api-access-rg928\") pod \"1b70ed28-93df-4bfc-9c1d-e992afc5bfc1\" (UID: \"1b70ed28-93df-4bfc-9c1d-e992afc5bfc1\") " Nov 04 04:51:24.510881 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:24.510847 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b70ed28-93df-4bfc-9c1d-e992afc5bfc1-bundle" pod "1b70ed28-93df-4bfc-9c1d-e992afc5bfc1" (UID: "1b70ed28-93df-4bfc-9c1d-e992afc5bfc1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:51:24.512621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:24.512591 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b70ed28-93df-4bfc-9c1d-e992afc5bfc1-kube-api-access-rg928" pod "1b70ed28-93df-4bfc-9c1d-e992afc5bfc1" (UID: "1b70ed28-93df-4bfc-9c1d-e992afc5bfc1"). InnerVolumeSpecName "kube-api-access-rg928". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:51:24.515134 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:24.515115 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b70ed28-93df-4bfc-9c1d-e992afc5bfc1-util" pod "1b70ed28-93df-4bfc-9c1d-e992afc5bfc1" (UID: "1b70ed28-93df-4bfc-9c1d-e992afc5bfc1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:51:24.611317 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:24.611286 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg928\" (UniqueName: \"kubernetes.io/projected/1b70ed28-93df-4bfc-9c1d-e992afc5bfc1-kube-api-access-rg928\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:51:24.611317 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:24.611311 4581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b70ed28-93df-4bfc-9c1d-e992afc5bfc1-bundle\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:51:24.611317 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:24.611320 4581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b70ed28-93df-4bfc-9c1d-e992afc5bfc1-util\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:51:25.309419 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:25.309397 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s" Nov 04 04:51:25.309419 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:25.309410 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96m7s" event={"ID":"1b70ed28-93df-4bfc-9c1d-e992afc5bfc1","Type":"ContainerDied","Data":"734b38cba0e1dcf1f27e2e0aa7021944e6bbb2bab2fa7eabe89d92c2bfb2cbe5"} Nov 04 04:51:25.309890 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:25.309450 4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="734b38cba0e1dcf1f27e2e0aa7021944e6bbb2bab2fa7eabe89d92c2bfb2cbe5" Nov 04 04:51:30.009627 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:30.009586 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:30.881607 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:30.881567 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:33.809694 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:33.809652 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:33.938051 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:33.938014 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:36.599106 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:36.599071 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:37.612587 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:37.612557 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:38.335580 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:38.335521 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:38.645655 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:38.645617 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:38.845543 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:38.845511 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:39.185480 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185453 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-229hq"] Nov 04 04:51:39.185757 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185747 4581 cpu_manager.go:414] "containerMap: removing container" podUID="1b70ed28-93df-4bfc-9c1d-e992afc5bfc1" containerName="extract" Nov 04 04:51:39.185808 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185758 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b70ed28-93df-4bfc-9c1d-e992afc5bfc1" containerName="extract" Nov 04 04:51:39.185808 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185770 4581 cpu_manager.go:414] "containerMap: removing container" podUID="1b70ed28-93df-4bfc-9c1d-e992afc5bfc1" containerName="pull" Nov 04 04:51:39.185808 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185775 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b70ed28-93df-4bfc-9c1d-e992afc5bfc1" containerName="pull" Nov 04 04:51:39.185808 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185795 4581 cpu_manager.go:414] "containerMap: removing container" podUID="06f04549-d6d9-4916-91b2-c8e97c52e716" containerName="extract" Nov 04 04:51:39.185808 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185801 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06f04549-d6d9-4916-91b2-c8e97c52e716" containerName="extract" Nov 04 04:51:39.185976 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185811 4581 cpu_manager.go:414] "containerMap: removing container" podUID="1b70ed28-93df-4bfc-9c1d-e992afc5bfc1" containerName="util" Nov 04 04:51:39.185976 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185818 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b70ed28-93df-4bfc-9c1d-e992afc5bfc1" containerName="util" Nov 04 04:51:39.185976 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185824 4581 cpu_manager.go:414] "containerMap: removing container" podUID="30217920-93bc-4930-92d2-b2e064730a3d" containerName="extract" Nov 04 04:51:39.185976 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185829 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30217920-93bc-4930-92d2-b2e064730a3d" containerName="extract" Nov 04 04:51:39.185976 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185838 4581 cpu_manager.go:414] "containerMap: removing container" podUID="2fa61466-9f70-403a-ab95-149e7a9a30de" containerName="pull" Nov 04 04:51:39.185976 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185843 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fa61466-9f70-403a-ab95-149e7a9a30de" containerName="pull" Nov 04 04:51:39.185976 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185848 4581 cpu_manager.go:414] "containerMap: removing container" podUID="2fa61466-9f70-403a-ab95-149e7a9a30de" containerName="extract" Nov 04 04:51:39.185976 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185853 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fa61466-9f70-403a-ab95-149e7a9a30de" containerName="extract" Nov 04 04:51:39.185976 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185863 4581 cpu_manager.go:414] "containerMap: removing container" podUID="30217920-93bc-4930-92d2-b2e064730a3d" containerName="pull" Nov 04 04:51:39.185976 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185867 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30217920-93bc-4930-92d2-b2e064730a3d" containerName="pull" Nov 04 04:51:39.185976 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185873 4581 cpu_manager.go:414] "containerMap: removing container" podUID="2fa61466-9f70-403a-ab95-149e7a9a30de" containerName="util" Nov 04 04:51:39.185976 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185877 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fa61466-9f70-403a-ab95-149e7a9a30de" containerName="util" Nov 04 04:51:39.185976 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185885 4581 cpu_manager.go:414] "containerMap: removing container" podUID="06f04549-d6d9-4916-91b2-c8e97c52e716" containerName="pull" Nov 04 04:51:39.185976 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185888 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06f04549-d6d9-4916-91b2-c8e97c52e716" containerName="pull" Nov 04 04:51:39.185976 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185898 4581 cpu_manager.go:414] "containerMap: removing container" podUID="06f04549-d6d9-4916-91b2-c8e97c52e716" containerName="util" Nov 04 04:51:39.185976 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185902 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06f04549-d6d9-4916-91b2-c8e97c52e716" containerName="util" Nov 04 04:51:39.185976 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185912 4581 cpu_manager.go:414] "containerMap: removing container" podUID="30217920-93bc-4930-92d2-b2e064730a3d" containerName="util" Nov 04 04:51:39.185976 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185916 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30217920-93bc-4930-92d2-b2e064730a3d" containerName="util" Nov 04 04:51:39.185976 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185955 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="06f04549-d6d9-4916-91b2-c8e97c52e716" containerName="extract" Nov 04 04:51:39.185976 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185964 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="30217920-93bc-4930-92d2-b2e064730a3d" containerName="extract" Nov 04 04:51:39.185976 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185970 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fa61466-9f70-403a-ab95-149e7a9a30de" containerName="extract" Nov 04 04:51:39.185976 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.185979 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b70ed28-93df-4bfc-9c1d-e992afc5bfc1" containerName="extract" Nov 04 04:51:39.187701 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.187685 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-229hq" Nov 04 04:51:39.251772 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.251741 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-229hq"] Nov 04 04:51:39.299637 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.299607 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 04 04:51:39.299787 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.299644 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 04 04:51:39.299787 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.299709 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-g25xt\"" Nov 04 04:51:39.334024 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.333980 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f57nf\" (UniqueName: \"kubernetes.io/projected/6d267c86-6d89-4718-a5f5-4d1f75db9adc-kube-api-access-f57nf\") pod \"cert-manager-webhook-5dbbbf75cc-229hq\" (UID: \"6d267c86-6d89-4718-a5f5-4d1f75db9adc\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-229hq" Nov 04 04:51:39.334024 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.334027 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d267c86-6d89-4718-a5f5-4d1f75db9adc-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-229hq\" (UID: \"6d267c86-6d89-4718-a5f5-4d1f75db9adc\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-229hq" Nov 04 04:51:39.435070 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.435044 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f57nf\" (UniqueName: \"kubernetes.io/projected/6d267c86-6d89-4718-a5f5-4d1f75db9adc-kube-api-access-f57nf\") pod \"cert-manager-webhook-5dbbbf75cc-229hq\" (UID: \"6d267c86-6d89-4718-a5f5-4d1f75db9adc\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-229hq" Nov 04 04:51:39.435070 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.435076 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d267c86-6d89-4718-a5f5-4d1f75db9adc-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-229hq\" (UID: \"6d267c86-6d89-4718-a5f5-4d1f75db9adc\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-229hq" Nov 04 04:51:39.492312 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.492246 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d267c86-6d89-4718-a5f5-4d1f75db9adc-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-229hq\" (UID: \"6d267c86-6d89-4718-a5f5-4d1f75db9adc\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-229hq" Nov 04 04:51:39.492312 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.492304 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f57nf\" (UniqueName: \"kubernetes.io/projected/6d267c86-6d89-4718-a5f5-4d1f75db9adc-kube-api-access-f57nf\") pod \"cert-manager-webhook-5dbbbf75cc-229hq\" (UID: \"6d267c86-6d89-4718-a5f5-4d1f75db9adc\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-229hq" Nov 04 04:51:39.508158 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.508137 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-229hq" Nov 04 04:51:39.875882 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:39.875852 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-229hq"] Nov 04 04:51:39.879238 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:51:39.879213 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d267c86_6d89_4718_a5f5_4d1f75db9adc.slice/crio-92e947ed0885406cfa3308e02f189521140764951fa0f7cbd0252742195dde17 WatchSource:0}: Error finding container 92e947ed0885406cfa3308e02f189521140764951fa0f7cbd0252742195dde17: Status 404 returned error can't find the container with id 92e947ed0885406cfa3308e02f189521140764951fa0f7cbd0252742195dde17 Nov 04 04:51:40.372581 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:40.372546 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-229hq" event={"ID":"6d267c86-6d89-4718-a5f5-4d1f75db9adc","Type":"ContainerStarted","Data":"92e947ed0885406cfa3308e02f189521140764951fa0f7cbd0252742195dde17"} Nov 04 04:51:40.659554 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:40.659467 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:41.149650 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:41.149614 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:42.218234 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:42.218198 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-6rlpl"] Nov 04 04:51:42.220676 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:42.220653 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6rlpl" Nov 04 04:51:42.273183 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:42.273157 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-9bh92\"" Nov 04 04:51:42.278007 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:42.277968 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-6rlpl"] Nov 04 04:51:42.362561 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:42.362524 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f83cfbfa-3926-43c4-ba05-c91bab1b0a09-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-6rlpl\" (UID: \"f83cfbfa-3926-43c4-ba05-c91bab1b0a09\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6rlpl" Nov 04 04:51:42.362755 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:42.362588 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqbn5\" (UniqueName: \"kubernetes.io/projected/f83cfbfa-3926-43c4-ba05-c91bab1b0a09-kube-api-access-zqbn5\") pod \"cert-manager-cainjector-5c57bbd4fc-6rlpl\" (UID: \"f83cfbfa-3926-43c4-ba05-c91bab1b0a09\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6rlpl" Nov 04 04:51:42.417833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:42.417797 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:42.463896 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:42.463856 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f83cfbfa-3926-43c4-ba05-c91bab1b0a09-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-6rlpl\" (UID: \"f83cfbfa-3926-43c4-ba05-c91bab1b0a09\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6rlpl" Nov 04 04:51:42.464102 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:42.463925 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqbn5\" (UniqueName: \"kubernetes.io/projected/f83cfbfa-3926-43c4-ba05-c91bab1b0a09-kube-api-access-zqbn5\") pod \"cert-manager-cainjector-5c57bbd4fc-6rlpl\" (UID: \"f83cfbfa-3926-43c4-ba05-c91bab1b0a09\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6rlpl" Nov 04 04:51:42.526910 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:42.526846 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f83cfbfa-3926-43c4-ba05-c91bab1b0a09-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-6rlpl\" (UID: \"f83cfbfa-3926-43c4-ba05-c91bab1b0a09\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6rlpl" Nov 04 04:51:42.528246 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:42.528222 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqbn5\" (UniqueName: \"kubernetes.io/projected/f83cfbfa-3926-43c4-ba05-c91bab1b0a09-kube-api-access-zqbn5\") pod \"cert-manager-cainjector-5c57bbd4fc-6rlpl\" (UID: \"f83cfbfa-3926-43c4-ba05-c91bab1b0a09\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6rlpl" Nov 04 04:51:42.534880 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:42.534858 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6rlpl" Nov 04 04:51:42.601963 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:42.601891 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:42.906286 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:42.906255 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-6rlpl"] Nov 04 04:51:42.909578 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:51:42.909545 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf83cfbfa_3926_43c4_ba05_c91bab1b0a09.slice/crio-9974312c4aeac851ac8429e9ebbc9a970acc145308132fb474635aa4be6cf002 WatchSource:0}: Error finding container 9974312c4aeac851ac8429e9ebbc9a970acc145308132fb474635aa4be6cf002: Status 404 returned error can't find the container with id 9974312c4aeac851ac8429e9ebbc9a970acc145308132fb474635aa4be6cf002 Nov 04 04:51:43.386455 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:43.386404 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6rlpl" event={"ID":"f83cfbfa-3926-43c4-ba05-c91bab1b0a09","Type":"ContainerStarted","Data":"9974312c4aeac851ac8429e9ebbc9a970acc145308132fb474635aa4be6cf002"} Nov 04 04:51:44.392434 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:44.392401 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6rlpl" event={"ID":"f83cfbfa-3926-43c4-ba05-c91bab1b0a09","Type":"ContainerStarted","Data":"0a98f9a2d3edcda15eec3b3cc0f2dc689de8acc4cb325565e8c2059aa9c151d5"} Nov 04 04:51:44.394146 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:44.394111 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-229hq" event={"ID":"6d267c86-6d89-4718-a5f5-4d1f75db9adc","Type":"ContainerStarted","Data":"3d425216c4f86ada15d61a8c7e81a9e3de66b9fc4296fa7b0148da07071b8b31"} Nov 04 04:51:44.394303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:44.394188 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-229hq" Nov 04 04:51:44.507450 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:44.507397 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6rlpl" podStartSLOduration=1.111175916 podStartE2EDuration="2.507384017s" totalImagesPullingTime="1.396208101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:51:44.506121206 +0000 UTC m=+263.396978221" watchObservedRunningTime="2026-11-04 04:51:44.507384017 +0000 UTC m=+263.398241037" Nov 04 04:51:44.620230 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:44.620168 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-229hq" podStartSLOduration=2.009403349 podStartE2EDuration="5.620151855s" totalImagesPullingTime="3.610748506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:51:44.619426528 +0000 UTC m=+263.510283548" watchObservedRunningTime="2026-11-04 04:51:44.620151855 +0000 UTC m=+263.511008874" Nov 04 04:51:45.536301 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:45.536268 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-kzkqf"] Nov 04 04:51:45.539837 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:45.539822 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kzkqf" Nov 04 04:51:45.598212 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:45.598174 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-qzsdv\"" Nov 04 04:51:45.647682 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:45.647656 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 04 04:51:45.647833 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:45.647720 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 04 04:51:45.649168 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:45.649149 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-kzkqf"] Nov 04 04:51:45.692194 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:45.692166 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4e6791a-09c6-456e-b5e4-f7d7ac67c908-tmp\") pod \"jobset-operator-674cd59dc5-kzkqf\" (UID: \"a4e6791a-09c6-456e-b5e4-f7d7ac67c908\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kzkqf" Nov 04 04:51:45.692311 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:45.692203 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgwh5\" (UniqueName: \"kubernetes.io/projected/a4e6791a-09c6-456e-b5e4-f7d7ac67c908-kube-api-access-hgwh5\") pod \"jobset-operator-674cd59dc5-kzkqf\" (UID: \"a4e6791a-09c6-456e-b5e4-f7d7ac67c908\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kzkqf" Nov 04 04:51:45.793074 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:45.793005 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4e6791a-09c6-456e-b5e4-f7d7ac67c908-tmp\") pod \"jobset-operator-674cd59dc5-kzkqf\" (UID: \"a4e6791a-09c6-456e-b5e4-f7d7ac67c908\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kzkqf" Nov 04 04:51:45.793074 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:45.793045 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgwh5\" (UniqueName: \"kubernetes.io/projected/a4e6791a-09c6-456e-b5e4-f7d7ac67c908-kube-api-access-hgwh5\") pod \"jobset-operator-674cd59dc5-kzkqf\" (UID: \"a4e6791a-09c6-456e-b5e4-f7d7ac67c908\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kzkqf" Nov 04 04:51:45.793315 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:45.793298 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4e6791a-09c6-456e-b5e4-f7d7ac67c908-tmp\") pod \"jobset-operator-674cd59dc5-kzkqf\" (UID: \"a4e6791a-09c6-456e-b5e4-f7d7ac67c908\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kzkqf" Nov 04 04:51:45.852185 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:45.852151 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgwh5\" (UniqueName: \"kubernetes.io/projected/a4e6791a-09c6-456e-b5e4-f7d7ac67c908-kube-api-access-hgwh5\") pod \"jobset-operator-674cd59dc5-kzkqf\" (UID: \"a4e6791a-09c6-456e-b5e4-f7d7ac67c908\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kzkqf" Nov 04 04:51:46.151062 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:46.151031 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kzkqf" Nov 04 04:51:46.460873 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:46.460790 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:46.537025 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:46.536912 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-kzkqf"] Nov 04 04:51:46.539566 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:51:46.539538 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4e6791a_09c6_456e_b5e4_f7d7ac67c908.slice/crio-70c7e6b90f19de7948a40727ebe9b7e8d7398634205793529718527ccc9bb58e WatchSource:0}: Error finding container 70c7e6b90f19de7948a40727ebe9b7e8d7398634205793529718527ccc9bb58e: Status 404 returned error can't find the container with id 70c7e6b90f19de7948a40727ebe9b7e8d7398634205793529718527ccc9bb58e Nov 04 04:51:46.659651 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:46.659620 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:46.818398 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:46.818299 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:47.407743 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:47.407707 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kzkqf" event={"ID":"a4e6791a-09c6-456e-b5e4-f7d7ac67c908","Type":"ContainerStarted","Data":"70c7e6b90f19de7948a40727ebe9b7e8d7398634205793529718527ccc9bb58e"} Nov 04 04:51:47.884790 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:51:47.884756 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 04:51:48.021209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:48.021179 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:48.297105 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:48.297014 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:48.558018 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:48.557898 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:48.685961 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:48.685923 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:48.931256 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:48.931224 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-4dcld"] Nov 04 04:51:48.937591 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:48.937570 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-4dcld" Nov 04 04:51:48.993321 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:48.993284 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-4dcld"] Nov 04 04:51:49.069715 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.069680 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:49.122605 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.122568 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4tk4\" (UniqueName: \"kubernetes.io/projected/ef22921e-77f0-4fd4-985d-5e909f32d3f7-kube-api-access-g4tk4\") pod \"cert-manager-597b95cb84-4dcld\" (UID: \"ef22921e-77f0-4fd4-985d-5e909f32d3f7\") " pod="cert-manager/cert-manager-597b95cb84-4dcld" Nov 04 04:51:49.122769 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.122638 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef22921e-77f0-4fd4-985d-5e909f32d3f7-bound-sa-token\") pod \"cert-manager-597b95cb84-4dcld\" (UID: \"ef22921e-77f0-4fd4-985d-5e909f32d3f7\") " pod="cert-manager/cert-manager-597b95cb84-4dcld" Nov 04 04:51:49.223265 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.223176 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4tk4\" (UniqueName: \"kubernetes.io/projected/ef22921e-77f0-4fd4-985d-5e909f32d3f7-kube-api-access-g4tk4\") pod \"cert-manager-597b95cb84-4dcld\" (UID: \"ef22921e-77f0-4fd4-985d-5e909f32d3f7\") " pod="cert-manager/cert-manager-597b95cb84-4dcld" Nov 04 04:51:49.223265 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.223239 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef22921e-77f0-4fd4-985d-5e909f32d3f7-bound-sa-token\") pod \"cert-manager-597b95cb84-4dcld\" (UID: \"ef22921e-77f0-4fd4-985d-5e909f32d3f7\") " pod="cert-manager/cert-manager-597b95cb84-4dcld" Nov 04 04:51:49.226504 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.226464 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:49.274351 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.274321 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq"] Nov 04 04:51:49.281315 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.281291 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" Nov 04 04:51:49.295072 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.295044 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef22921e-77f0-4fd4-985d-5e909f32d3f7-bound-sa-token\") pod \"cert-manager-597b95cb84-4dcld\" (UID: \"ef22921e-77f0-4fd4-985d-5e909f32d3f7\") " pod="cert-manager/cert-manager-597b95cb84-4dcld" Nov 04 04:51:49.295528 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.295501 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4tk4\" (UniqueName: \"kubernetes.io/projected/ef22921e-77f0-4fd4-985d-5e909f32d3f7-kube-api-access-g4tk4\") pod \"cert-manager-597b95cb84-4dcld\" (UID: \"ef22921e-77f0-4fd4-985d-5e909f32d3f7\") " pod="cert-manager/cert-manager-597b95cb84-4dcld" Nov 04 04:51:49.336173 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.336138 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 04 04:51:49.336333 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.336202 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 04 04:51:49.371284 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.371252 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 04 04:51:49.371462 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.371407 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 04 04:51:49.371462 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.371447 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 04 04:51:49.371552 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.371463 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-nj9j7\"" Nov 04 04:51:49.382192 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.382155 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:49.420277 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.420248 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq"] Nov 04 04:51:49.425532 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.425505 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/caa1a5cf-cfb7-47e3-a013-5110c1c130a9-metrics-cert\") pod \"opendatahub-operator-controller-manager-84f7664b48-gzcdq\" (UID: \"caa1a5cf-cfb7-47e3-a013-5110c1c130a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" Nov 04 04:51:49.425696 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.425541 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdlhr\" (UniqueName: \"kubernetes.io/projected/caa1a5cf-cfb7-47e3-a013-5110c1c130a9-kube-api-access-wdlhr\") pod \"opendatahub-operator-controller-manager-84f7664b48-gzcdq\" (UID: \"caa1a5cf-cfb7-47e3-a013-5110c1c130a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" Nov 04 04:51:49.425696 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.425570 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/caa1a5cf-cfb7-47e3-a013-5110c1c130a9-webhook-cert\") pod \"opendatahub-operator-controller-manager-84f7664b48-gzcdq\" (UID: \"caa1a5cf-cfb7-47e3-a013-5110c1c130a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" Nov 04 04:51:49.425819 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.425732 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/caa1a5cf-cfb7-47e3-a013-5110c1c130a9-tmp\") pod \"opendatahub-operator-controller-manager-84f7664b48-gzcdq\" (UID: \"caa1a5cf-cfb7-47e3-a013-5110c1c130a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" Nov 04 04:51:49.425819 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.425772 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/caa1a5cf-cfb7-47e3-a013-5110c1c130a9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84f7664b48-gzcdq\" (UID: \"caa1a5cf-cfb7-47e3-a013-5110c1c130a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" Nov 04 04:51:49.425924 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.425839 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/caa1a5cf-cfb7-47e3-a013-5110c1c130a9-manifests\") pod \"opendatahub-operator-controller-manager-84f7664b48-gzcdq\" (UID: \"caa1a5cf-cfb7-47e3-a013-5110c1c130a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" Nov 04 04:51:49.527369 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.527288 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/caa1a5cf-cfb7-47e3-a013-5110c1c130a9-tmp\") pod \"opendatahub-operator-controller-manager-84f7664b48-gzcdq\" (UID: \"caa1a5cf-cfb7-47e3-a013-5110c1c130a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" Nov 04 04:51:49.527369 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.527341 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/caa1a5cf-cfb7-47e3-a013-5110c1c130a9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84f7664b48-gzcdq\" (UID: \"caa1a5cf-cfb7-47e3-a013-5110c1c130a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" Nov 04 04:51:49.527607 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.527383 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/caa1a5cf-cfb7-47e3-a013-5110c1c130a9-manifests\") pod \"opendatahub-operator-controller-manager-84f7664b48-gzcdq\" (UID: \"caa1a5cf-cfb7-47e3-a013-5110c1c130a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" Nov 04 04:51:49.527607 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.527423 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/caa1a5cf-cfb7-47e3-a013-5110c1c130a9-metrics-cert\") pod \"opendatahub-operator-controller-manager-84f7664b48-gzcdq\" (UID: \"caa1a5cf-cfb7-47e3-a013-5110c1c130a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" Nov 04 04:51:49.527607 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.527442 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdlhr\" (UniqueName: \"kubernetes.io/projected/caa1a5cf-cfb7-47e3-a013-5110c1c130a9-kube-api-access-wdlhr\") pod \"opendatahub-operator-controller-manager-84f7664b48-gzcdq\" (UID: \"caa1a5cf-cfb7-47e3-a013-5110c1c130a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" Nov 04 04:51:49.527607 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.527472 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/caa1a5cf-cfb7-47e3-a013-5110c1c130a9-webhook-cert\") pod \"opendatahub-operator-controller-manager-84f7664b48-gzcdq\" (UID: \"caa1a5cf-cfb7-47e3-a013-5110c1c130a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" Nov 04 04:51:49.527859 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.527829 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/caa1a5cf-cfb7-47e3-a013-5110c1c130a9-manifests\") pod \"opendatahub-operator-controller-manager-84f7664b48-gzcdq\" (UID: \"caa1a5cf-cfb7-47e3-a013-5110c1c130a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" Nov 04 04:51:49.530132 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.530099 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/caa1a5cf-cfb7-47e3-a013-5110c1c130a9-tmp\") pod \"opendatahub-operator-controller-manager-84f7664b48-gzcdq\" (UID: \"caa1a5cf-cfb7-47e3-a013-5110c1c130a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" Nov 04 04:51:49.530273 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.530257 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/caa1a5cf-cfb7-47e3-a013-5110c1c130a9-metrics-cert\") pod \"opendatahub-operator-controller-manager-84f7664b48-gzcdq\" (UID: \"caa1a5cf-cfb7-47e3-a013-5110c1c130a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" Nov 04 04:51:49.530433 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.530417 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/caa1a5cf-cfb7-47e3-a013-5110c1c130a9-webhook-cert\") pod \"opendatahub-operator-controller-manager-84f7664b48-gzcdq\" (UID: \"caa1a5cf-cfb7-47e3-a013-5110c1c130a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" Nov 04 04:51:49.530433 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.530422 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/caa1a5cf-cfb7-47e3-a013-5110c1c130a9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84f7664b48-gzcdq\" (UID: \"caa1a5cf-cfb7-47e3-a013-5110c1c130a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" Nov 04 04:51:49.551472 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.551448 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-4dcld" Nov 04 04:51:49.569916 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.569878 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:49.590432 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.590358 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdlhr\" (UniqueName: \"kubernetes.io/projected/caa1a5cf-cfb7-47e3-a013-5110c1c130a9-kube-api-access-wdlhr\") pod \"opendatahub-operator-controller-manager-84f7664b48-gzcdq\" (UID: \"caa1a5cf-cfb7-47e3-a013-5110c1c130a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" Nov 04 04:51:49.599298 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.598947 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" Nov 04 04:51:49.747245 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.747202 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:49.892930 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.892897 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:49.941461 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:49.941378 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-4dcld"] Nov 04 04:51:49.944178 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:51:49.944149 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef22921e_77f0_4fd4_985d_5e909f32d3f7.slice/crio-a8730d57610712258a4fea9d58ac6d545f0b51fb16583d7856e2d792f28580cd WatchSource:0}: Error finding container a8730d57610712258a4fea9d58ac6d545f0b51fb16583d7856e2d792f28580cd: Status 404 returned error can't find the container with id a8730d57610712258a4fea9d58ac6d545f0b51fb16583d7856e2d792f28580cd Nov 04 04:51:50.063171 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:50.063134 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:50.090793 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:50.090215 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq"] Nov 04 04:51:50.093371 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:51:50.093277 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa1a5cf_cfb7_47e3_a013_5110c1c130a9.slice/crio-54ddf0788644d510da2cf6d07e3fe7b86d05aadc333648a4ea3b251129078646 WatchSource:0}: Error finding container 54ddf0788644d510da2cf6d07e3fe7b86d05aadc333648a4ea3b251129078646: Status 404 returned error can't find the container with id 54ddf0788644d510da2cf6d07e3fe7b86d05aadc333648a4ea3b251129078646 Nov 04 04:51:50.244251 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:50.244168 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:50.400548 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:50.400523 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-229hq" Nov 04 04:51:50.425323 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:50.425291 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" event={"ID":"caa1a5cf-cfb7-47e3-a013-5110c1c130a9","Type":"ContainerStarted","Data":"54ddf0788644d510da2cf6d07e3fe7b86d05aadc333648a4ea3b251129078646"} Nov 04 04:51:50.426691 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:50.426671 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-4dcld" event={"ID":"ef22921e-77f0-4fd4-985d-5e909f32d3f7","Type":"ContainerStarted","Data":"df55a0f71984b00afe3bd8e77c81be2c8eb6b2a1699b3b72d087e077ebe54337"} Nov 04 04:51:50.426749 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:50.426704 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-4dcld" event={"ID":"ef22921e-77f0-4fd4-985d-5e909f32d3f7","Type":"ContainerStarted","Data":"a8730d57610712258a4fea9d58ac6d545f0b51fb16583d7856e2d792f28580cd"} Nov 04 04:51:50.638534 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:50.638489 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-4dcld" podStartSLOduration=2.6384755650000002 podStartE2EDuration="2.638475565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:51:50.636522859 +0000 UTC m=+269.527379880" watchObservedRunningTime="2026-11-04 04:51:50.638475565 +0000 UTC m=+269.529332584" Nov 04 04:51:51.034572 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:51.034486 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:51.258239 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:51.258185 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:51.437337 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:51.437301 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kzkqf" event={"ID":"a4e6791a-09c6-456e-b5e4-f7d7ac67c908","Type":"ContainerStarted","Data":"0cf2e62bc39bd7ed50e77bc2c123b907d93c4b5fb8e8d1dbff54ada62663c036"} Nov 04 04:51:51.576298 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:51.557235 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kzkqf" podStartSLOduration=2.480116758 podStartE2EDuration="6.557220252s" totalImagesPullingTime="4.077103494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:51:51.554898262 +0000 UTC m=+270.445755283" watchObservedRunningTime="2026-11-04 04:51:51.557220252 +0000 UTC m=+270.448077270" Nov 04 04:51:53.319193 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:53.319156 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:54.452281 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:54.452249 4581 generic.go:356] "Generic (PLEG): container finished" podID="caa1a5cf-cfb7-47e3-a013-5110c1c130a9" containerID="91fe01eb681ae3efd56fac441ce284d7ee15c0384cece60300ab0de79c59c4b2" exitCode=0 Nov 04 04:51:54.452687 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:54.452344 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" event={"ID":"caa1a5cf-cfb7-47e3-a013-5110c1c130a9","Type":"ContainerDied","Data":"91fe01eb681ae3efd56fac441ce284d7ee15c0384cece60300ab0de79c59c4b2"} Nov 04 04:51:54.611113 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:54.611032 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:55.217039 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:55.217010 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:55.458740 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:55.458695 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" event={"ID":"caa1a5cf-cfb7-47e3-a013-5110c1c130a9","Type":"ContainerStarted","Data":"4dc0e8f4a3fdc342653e49892be6a309b22ad5819260911bfdb73c304eb2eb9a"} Nov 04 04:51:55.459205 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:55.458801 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" Nov 04 04:51:55.641407 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:55.641355 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" podStartSLOduration=1.6692650599999999 podStartE2EDuration="6.641330949s" totalImagesPullingTime="4.492048828s" totalInitContainerRuntime="480.017061ms" isStatefulPod=false podCreationTimestamp="2026-11-04 04:51:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:51:55.589472639 +0000 UTC m=+274.480329659" watchObservedRunningTime="2026-11-04 04:51:55.641330949 +0000 UTC m=+274.532188034" Nov 04 04:51:56.516724 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:56.516691 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:56.565308 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:56.565274 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-4bhfh"] Nov 04 04:51:56.567863 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:56.567846 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4bhfh" Nov 04 04:51:56.584607 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:56.584576 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a19b0538-b992-48a4-8566-960d59838722-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-4bhfh\" (UID: \"a19b0538-b992-48a4-8566-960d59838722\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4bhfh" Nov 04 04:51:56.584712 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:56.584687 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwmdk\" (UniqueName: \"kubernetes.io/projected/a19b0538-b992-48a4-8566-960d59838722-kube-api-access-pwmdk\") pod \"kuadrant-console-plugin-66c4fb575d-4bhfh\" (UID: \"a19b0538-b992-48a4-8566-960d59838722\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4bhfh" Nov 04 04:51:56.584758 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:56.584718 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a19b0538-b992-48a4-8566-960d59838722-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-4bhfh\" (UID: \"a19b0538-b992-48a4-8566-960d59838722\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4bhfh" Nov 04 04:51:56.623601 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:56.623577 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 04 04:51:56.624109 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:56.624097 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 04 04:51:56.624194 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:56.624131 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 04 04:51:56.629096 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:56.629081 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-m7cqj\"" Nov 04 04:51:56.673030 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:56.673002 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 04 04:51:56.674715 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:56.674694 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-4bhfh"] Nov 04 04:51:56.685370 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:56.685348 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a19b0538-b992-48a4-8566-960d59838722-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-4bhfh\" (UID: \"a19b0538-b992-48a4-8566-960d59838722\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4bhfh" Nov 04 04:51:56.685471 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:56.685406 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwmdk\" (UniqueName: \"kubernetes.io/projected/a19b0538-b992-48a4-8566-960d59838722-kube-api-access-pwmdk\") pod \"kuadrant-console-plugin-66c4fb575d-4bhfh\" (UID: \"a19b0538-b992-48a4-8566-960d59838722\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4bhfh" Nov 04 04:51:56.685471 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:56.685433 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a19b0538-b992-48a4-8566-960d59838722-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-4bhfh\" (UID: \"a19b0538-b992-48a4-8566-960d59838722\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4bhfh" Nov 04 04:51:56.686035 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:56.686016 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a19b0538-b992-48a4-8566-960d59838722-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-4bhfh\" (UID: \"a19b0538-b992-48a4-8566-960d59838722\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4bhfh" Nov 04 04:51:56.687892 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:56.687870 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a19b0538-b992-48a4-8566-960d59838722-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-4bhfh\" (UID: \"a19b0538-b992-48a4-8566-960d59838722\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4bhfh" Nov 04 04:51:56.739480 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:56.739451 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:56.743450 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:56.743424 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwmdk\" (UniqueName: \"kubernetes.io/projected/a19b0538-b992-48a4-8566-960d59838722-kube-api-access-pwmdk\") pod \"kuadrant-console-plugin-66c4fb575d-4bhfh\" (UID: \"a19b0538-b992-48a4-8566-960d59838722\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4bhfh" Nov 04 04:51:56.878738 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:56.878712 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4bhfh" Nov 04 04:51:57.248075 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:57.248050 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-4bhfh"] Nov 04 04:51:57.250264 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:51:57.250232 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda19b0538_b992_48a4_8566_960d59838722.slice/crio-6db3da9a21a8483c91b217172f4e664f79dfd90329880be32c4bf571a1b81c54 WatchSource:0}: Error finding container 6db3da9a21a8483c91b217172f4e664f79dfd90329880be32c4bf571a1b81c54: Status 404 returned error can't find the container with id 6db3da9a21a8483c91b217172f4e664f79dfd90329880be32c4bf571a1b81c54 Nov 04 04:51:57.473778 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:57.473740 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4bhfh" event={"ID":"a19b0538-b992-48a4-8566-960d59838722","Type":"ContainerStarted","Data":"6db3da9a21a8483c91b217172f4e664f79dfd90329880be32c4bf571a1b81c54"} Nov 04 04:51:57.686932 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:57.686898 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:51:59.120168 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:51:59.120128 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:01.611424 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:01.611369 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:01.749085 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:01.749051 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:02.228440 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:02.228408 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:02.354082 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:02.354050 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:02.813796 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:02.813741 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:03.002604 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:03.002570 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:03.506704 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:03.506612 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4bhfh" event={"ID":"a19b0538-b992-48a4-8566-960d59838722","Type":"ContainerStarted","Data":"0dd338336967bd0e2ab32f6f88511e4ca2dddc29978fb97ec177dfed80b6fee6"} Nov 04 04:52:03.626460 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:03.626392 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4bhfh" podStartSLOduration=1.707217194 podStartE2EDuration="7.626374174s" totalImagesPullingTime="5.91915698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:51:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:52:03.626039689 +0000 UTC m=+282.516896710" watchObservedRunningTime="2026-11-04 04:52:03.626374174 +0000 UTC m=+282.517231196" Nov 04 04:52:06.467508 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:06.467477 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-gzcdq" Nov 04 04:52:09.616369 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:09.616332 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:09.791554 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:09.791521 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:13.435089 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:13.435043 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:13.610262 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:13.610231 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:17.885417 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:52:17.885385 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 04:52:18.244541 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:18.244443 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:18.386367 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:18.386307 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:18.530957 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:18.530861 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:18.959976 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:18.959922 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:20.308649 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:20.308613 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:20.452995 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:20.452963 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-546d78d5c-g8s92"] Nov 04 04:52:20.456845 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:20.456814 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:20.460330 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:20.460312 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-546d78d5c-g8s92" Nov 04 04:52:20.513423 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:20.513393 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 04 04:52:20.563952 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:20.563880 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-w6k6c\"" Nov 04 04:52:20.564183 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:20.564147 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 04 04:52:20.564183 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:20.564096 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 04 04:52:20.564183 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:20.564036 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 04 04:52:20.565558 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:20.565535 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-546d78d5c-g8s92"] Nov 04 04:52:20.602187 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:20.602153 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63a561a1-32d4-41b1-a285-4e82379ac8d4-tls-certs\") pod \"kube-auth-proxy-546d78d5c-g8s92\" (UID: \"63a561a1-32d4-41b1-a285-4e82379ac8d4\") " pod="openshift-ingress/kube-auth-proxy-546d78d5c-g8s92" Nov 04 04:52:20.602329 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:20.602304 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cs4b\" (UniqueName: \"kubernetes.io/projected/63a561a1-32d4-41b1-a285-4e82379ac8d4-kube-api-access-4cs4b\") pod \"kube-auth-proxy-546d78d5c-g8s92\" (UID: \"63a561a1-32d4-41b1-a285-4e82379ac8d4\") " pod="openshift-ingress/kube-auth-proxy-546d78d5c-g8s92" Nov 04 04:52:20.602394 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:20.602377 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63a561a1-32d4-41b1-a285-4e82379ac8d4-tmp\") pod \"kube-auth-proxy-546d78d5c-g8s92\" (UID: \"63a561a1-32d4-41b1-a285-4e82379ac8d4\") " pod="openshift-ingress/kube-auth-proxy-546d78d5c-g8s92" Nov 04 04:52:20.703208 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:20.703176 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cs4b\" (UniqueName: \"kubernetes.io/projected/63a561a1-32d4-41b1-a285-4e82379ac8d4-kube-api-access-4cs4b\") pod \"kube-auth-proxy-546d78d5c-g8s92\" (UID: \"63a561a1-32d4-41b1-a285-4e82379ac8d4\") " pod="openshift-ingress/kube-auth-proxy-546d78d5c-g8s92" Nov 04 04:52:20.703356 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:20.703215 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63a561a1-32d4-41b1-a285-4e82379ac8d4-tmp\") pod \"kube-auth-proxy-546d78d5c-g8s92\" (UID: \"63a561a1-32d4-41b1-a285-4e82379ac8d4\") " pod="openshift-ingress/kube-auth-proxy-546d78d5c-g8s92" Nov 04 04:52:20.703406 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:20.703345 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63a561a1-32d4-41b1-a285-4e82379ac8d4-tls-certs\") pod \"kube-auth-proxy-546d78d5c-g8s92\" (UID: \"63a561a1-32d4-41b1-a285-4e82379ac8d4\") " pod="openshift-ingress/kube-auth-proxy-546d78d5c-g8s92" Nov 04 04:52:20.705795 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:20.705763 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63a561a1-32d4-41b1-a285-4e82379ac8d4-tmp\") pod \"kube-auth-proxy-546d78d5c-g8s92\" (UID: \"63a561a1-32d4-41b1-a285-4e82379ac8d4\") " pod="openshift-ingress/kube-auth-proxy-546d78d5c-g8s92" Nov 04 04:52:20.705915 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:20.705899 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63a561a1-32d4-41b1-a285-4e82379ac8d4-tls-certs\") pod \"kube-auth-proxy-546d78d5c-g8s92\" (UID: \"63a561a1-32d4-41b1-a285-4e82379ac8d4\") " pod="openshift-ingress/kube-auth-proxy-546d78d5c-g8s92" Nov 04 04:52:20.762933 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:20.762905 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cs4b\" (UniqueName: \"kubernetes.io/projected/63a561a1-32d4-41b1-a285-4e82379ac8d4-kube-api-access-4cs4b\") pod \"kube-auth-proxy-546d78d5c-g8s92\" (UID: \"63a561a1-32d4-41b1-a285-4e82379ac8d4\") " pod="openshift-ingress/kube-auth-proxy-546d78d5c-g8s92" Nov 04 04:52:20.772768 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:20.772747 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-546d78d5c-g8s92" Nov 04 04:52:20.900743 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:20.900710 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:21.027275 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:21.027243 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:21.145856 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:21.145835 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-546d78d5c-g8s92"] Nov 04 04:52:21.148397 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:52:21.148369 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a561a1_32d4_41b1_a285_4e82379ac8d4.slice/crio-f1b50f935dd30f99036f1bf27de3e6f7715aa2df604c96dd9894abf543b49475 WatchSource:0}: Error finding container f1b50f935dd30f99036f1bf27de3e6f7715aa2df604c96dd9894abf543b49475: Status 404 returned error can't find the container with id f1b50f935dd30f99036f1bf27de3e6f7715aa2df604c96dd9894abf543b49475 Nov 04 04:52:21.177309 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:21.177257 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:21.314812 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:21.314776 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:21.453497 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:21.453387 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:21.587621 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:21.587590 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-546d78d5c-g8s92" event={"ID":"63a561a1-32d4-41b1-a285-4e82379ac8d4","Type":"ContainerStarted","Data":"f1b50f935dd30f99036f1bf27de3e6f7715aa2df604c96dd9894abf543b49475"} Nov 04 04:52:23.116052 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:23.116016 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:23.252862 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:23.252824 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:23.948895 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:23.948859 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d8459b48-s9hm4"] Nov 04 04:52:23.953940 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:23.953911 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5d8459b48-s9hm4" Nov 04 04:52:24.040761 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:24.040728 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91cd473d-0e1c-4dbc-a2a2-d5719a35ded4-metrics-certs\") pod \"jobset-controller-manager-5d8459b48-s9hm4\" (UID: \"91cd473d-0e1c-4dbc-a2a2-d5719a35ded4\") " pod="openshift-jobset-operator/jobset-controller-manager-5d8459b48-s9hm4" Nov 04 04:52:24.040884 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:24.040783 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfdn2\" (UniqueName: \"kubernetes.io/projected/91cd473d-0e1c-4dbc-a2a2-d5719a35ded4-kube-api-access-tfdn2\") pod \"jobset-controller-manager-5d8459b48-s9hm4\" (UID: \"91cd473d-0e1c-4dbc-a2a2-d5719a35ded4\") " pod="openshift-jobset-operator/jobset-controller-manager-5d8459b48-s9hm4" Nov 04 04:52:24.040884 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:24.040859 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91cd473d-0e1c-4dbc-a2a2-d5719a35ded4-cert\") pod \"jobset-controller-manager-5d8459b48-s9hm4\" (UID: \"91cd473d-0e1c-4dbc-a2a2-d5719a35ded4\") " pod="openshift-jobset-operator/jobset-controller-manager-5d8459b48-s9hm4" Nov 04 04:52:24.041021 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:24.040909 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/91cd473d-0e1c-4dbc-a2a2-d5719a35ded4-manager-config\") pod \"jobset-controller-manager-5d8459b48-s9hm4\" (UID: \"91cd473d-0e1c-4dbc-a2a2-d5719a35ded4\") " pod="openshift-jobset-operator/jobset-controller-manager-5d8459b48-s9hm4" Nov 04 04:52:24.058252 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:24.058228 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 04 04:52:24.058372 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:24.058365 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-f9lgz\"" Nov 04 04:52:24.058439 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:24.058400 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 04 04:52:24.058439 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:24.058415 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 04 04:52:24.059908 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:24.059890 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d8459b48-s9hm4"] Nov 04 04:52:24.086089 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:24.086068 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 04 04:52:24.142054 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:24.142030 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91cd473d-0e1c-4dbc-a2a2-d5719a35ded4-metrics-certs\") pod \"jobset-controller-manager-5d8459b48-s9hm4\" (UID: \"91cd473d-0e1c-4dbc-a2a2-d5719a35ded4\") " pod="openshift-jobset-operator/jobset-controller-manager-5d8459b48-s9hm4" Nov 04 04:52:24.142356 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:24.142086 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfdn2\" (UniqueName: \"kubernetes.io/projected/91cd473d-0e1c-4dbc-a2a2-d5719a35ded4-kube-api-access-tfdn2\") pod \"jobset-controller-manager-5d8459b48-s9hm4\" (UID: \"91cd473d-0e1c-4dbc-a2a2-d5719a35ded4\") " pod="openshift-jobset-operator/jobset-controller-manager-5d8459b48-s9hm4" Nov 04 04:52:24.142356 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:24.142221 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91cd473d-0e1c-4dbc-a2a2-d5719a35ded4-cert\") pod \"jobset-controller-manager-5d8459b48-s9hm4\" (UID: \"91cd473d-0e1c-4dbc-a2a2-d5719a35ded4\") " pod="openshift-jobset-operator/jobset-controller-manager-5d8459b48-s9hm4" Nov 04 04:52:24.142356 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:24.142303 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/91cd473d-0e1c-4dbc-a2a2-d5719a35ded4-manager-config\") pod \"jobset-controller-manager-5d8459b48-s9hm4\" (UID: \"91cd473d-0e1c-4dbc-a2a2-d5719a35ded4\") " pod="openshift-jobset-operator/jobset-controller-manager-5d8459b48-s9hm4" Nov 04 04:52:24.142979 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:24.142957 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/91cd473d-0e1c-4dbc-a2a2-d5719a35ded4-manager-config\") pod \"jobset-controller-manager-5d8459b48-s9hm4\" (UID: \"91cd473d-0e1c-4dbc-a2a2-d5719a35ded4\") " pod="openshift-jobset-operator/jobset-controller-manager-5d8459b48-s9hm4" Nov 04 04:52:24.144543 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:24.144516 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91cd473d-0e1c-4dbc-a2a2-d5719a35ded4-metrics-certs\") pod \"jobset-controller-manager-5d8459b48-s9hm4\" (UID: \"91cd473d-0e1c-4dbc-a2a2-d5719a35ded4\") " pod="openshift-jobset-operator/jobset-controller-manager-5d8459b48-s9hm4" Nov 04 04:52:24.144658 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:24.144641 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91cd473d-0e1c-4dbc-a2a2-d5719a35ded4-cert\") pod \"jobset-controller-manager-5d8459b48-s9hm4\" (UID: \"91cd473d-0e1c-4dbc-a2a2-d5719a35ded4\") " pod="openshift-jobset-operator/jobset-controller-manager-5d8459b48-s9hm4" Nov 04 04:52:24.203010 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:24.202920 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfdn2\" (UniqueName: \"kubernetes.io/projected/91cd473d-0e1c-4dbc-a2a2-d5719a35ded4-kube-api-access-tfdn2\") pod \"jobset-controller-manager-5d8459b48-s9hm4\" (UID: \"91cd473d-0e1c-4dbc-a2a2-d5719a35ded4\") " pod="openshift-jobset-operator/jobset-controller-manager-5d8459b48-s9hm4" Nov 04 04:52:24.269760 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:24.269714 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5d8459b48-s9hm4" Nov 04 04:52:24.603217 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:24.603187 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-546d78d5c-g8s92" event={"ID":"63a561a1-32d4-41b1-a285-4e82379ac8d4","Type":"ContainerStarted","Data":"bc144aed2030f97e1db4245e34aeb4ab7214d612c15787ddf01a2b8311e01d74"} Nov 04 04:52:24.641057 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:24.640975 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d8459b48-s9hm4"] Nov 04 04:52:24.643227 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:52:24.643204 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91cd473d_0e1c_4dbc_a2a2_d5719a35ded4.slice/crio-a88bcc08c4a44adfa12ab6fccfb792aca81b14f6473d16e9224c7a8c8685830a WatchSource:0}: Error finding container a88bcc08c4a44adfa12ab6fccfb792aca81b14f6473d16e9224c7a8c8685830a: Status 404 returned error can't find the container with id a88bcc08c4a44adfa12ab6fccfb792aca81b14f6473d16e9224c7a8c8685830a Nov 04 04:52:24.645173 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:24.645153 4581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 04:52:24.717147 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:24.717098 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-546d78d5c-g8s92" podStartSLOduration=1.834121656 podStartE2EDuration="4.717083992s" totalImagesPullingTime="2.882962336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:52:24.716621783 +0000 UTC m=+303.607478804" watchObservedRunningTime="2026-11-04 04:52:24.717083992 +0000 UTC m=+303.607941011" Nov 04 04:52:25.609512 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:25.609470 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5d8459b48-s9hm4" event={"ID":"91cd473d-0e1c-4dbc-a2a2-d5719a35ded4","Type":"ContainerStarted","Data":"a88bcc08c4a44adfa12ab6fccfb792aca81b14f6473d16e9224c7a8c8685830a"} Nov 04 04:52:27.854878 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:27.854845 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:29.522065 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:29.522026 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:29.630775 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:29.630736 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5d8459b48-s9hm4" event={"ID":"91cd473d-0e1c-4dbc-a2a2-d5719a35ded4","Type":"ContainerStarted","Data":"0d2593f9ed2185ea01c4d6a2cf15d5a56e6e49148f1166a38c0e04c74b755c19"} Nov 04 04:52:29.630960 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:29.630941 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5d8459b48-s9hm4" Nov 04 04:52:29.704575 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:29.704519 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:29.746611 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:29.746557 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5d8459b48-s9hm4" podStartSLOduration=2.782723596 podStartE2EDuration="6.746542376s" totalImagesPullingTime="3.96381878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:52:29.745321239 +0000 UTC m=+308.636178256" watchObservedRunningTime="2026-11-04 04:52:29.746542376 +0000 UTC m=+308.637399396" Nov 04 04:52:34.267744 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:34.267657 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:34.441216 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:34.441184 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:38.950152 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:38.950111 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:39.154135 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:39.154094 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:39.312542 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:39.312448 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:39.452917 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:39.452883 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:40.640826 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:40.640798 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5d8459b48-s9hm4" Nov 04 04:52:40.651903 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:40.651870 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:40.779144 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:40.779111 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:44.272596 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:44.272552 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:44.392817 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:44.392785 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:44.677427 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:44.677385 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:44.800253 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:44.800219 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:44.935366 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:44.935270 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:47.885842 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:52:47.885808 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 04:52:53.282759 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:53.282722 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:53.401780 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:53.401741 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:57.268518 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:57.268483 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:57.400149 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:57.400114 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:52:58.026228 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:52:58.026191 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:00.366568 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:00.366528 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:02.081919 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:02.081880 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:02.217916 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:02.217879 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:02.669271 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:02.669242 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:02.849725 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:02.849693 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:06.624513 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:06.624474 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:06.755390 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:06.755354 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:07.400512 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:07.400481 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:13.130516 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:13.130479 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:13.303494 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:13.303462 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:17.886211 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:53:17.886174 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 04:53:28.264583 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:28.264540 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:28.440394 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:28.440366 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:28.578307 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:28.578221 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:28.711711 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:28.711672 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:32.990618 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:32.990580 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:33.117462 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:33.117429 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:36.192834 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:36.192788 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:36.421046 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:36.421011 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:47.885126 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:53:47.885086 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 04:53:50.135608 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:50.135573 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:50.313505 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:50.313474 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:50.436314 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:50.436230 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:50.566495 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:50.566463 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:50.687756 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:50.687674 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:50.816055 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:50.816014 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:50.936584 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:50.936550 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:51.062263 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:51.062178 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:51.187648 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:51.187603 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:51.346467 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:51.346392 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-qprw2"] Nov 04 04:53:51.349756 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:51.349741 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-qprw2" Nov 04 04:53:51.407552 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:51.407517 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-qprw2"] Nov 04 04:53:51.412766 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:51.412739 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa715875-4789-4276-8b9e-c809de6606b6-metrics-certs\") pod \"odh-model-controller-769875db56-qprw2\" (UID: \"fa715875-4789-4276-8b9e-c809de6606b6\") " pod="opendatahub/odh-model-controller-769875db56-qprw2" Nov 04 04:53:51.412906 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:51.412775 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa715875-4789-4276-8b9e-c809de6606b6-cert\") pod \"odh-model-controller-769875db56-qprw2\" (UID: \"fa715875-4789-4276-8b9e-c809de6606b6\") " pod="opendatahub/odh-model-controller-769875db56-qprw2" Nov 04 04:53:51.412906 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:51.412814 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqg2m\" (UniqueName: \"kubernetes.io/projected/fa715875-4789-4276-8b9e-c809de6606b6-kube-api-access-gqg2m\") pod \"odh-model-controller-769875db56-qprw2\" (UID: \"fa715875-4789-4276-8b9e-c809de6606b6\") " pod="opendatahub/odh-model-controller-769875db56-qprw2" Nov 04 04:53:51.455522 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:51.455488 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 04 04:53:51.455522 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:51.455502 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 04 04:53:51.455739 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:51.455535 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-6ncf8\"" Nov 04 04:53:51.455739 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:51.455620 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 04 04:53:51.455739 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:51.455685 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 04 04:53:51.513612 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:51.513582 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa715875-4789-4276-8b9e-c809de6606b6-metrics-certs\") pod \"odh-model-controller-769875db56-qprw2\" (UID: \"fa715875-4789-4276-8b9e-c809de6606b6\") " pod="opendatahub/odh-model-controller-769875db56-qprw2" Nov 04 04:53:51.513729 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:51.513614 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa715875-4789-4276-8b9e-c809de6606b6-cert\") pod \"odh-model-controller-769875db56-qprw2\" (UID: \"fa715875-4789-4276-8b9e-c809de6606b6\") " pod="opendatahub/odh-model-controller-769875db56-qprw2" Nov 04 04:53:51.513729 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:51.513643 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqg2m\" (UniqueName: \"kubernetes.io/projected/fa715875-4789-4276-8b9e-c809de6606b6-kube-api-access-gqg2m\") pod \"odh-model-controller-769875db56-qprw2\" (UID: \"fa715875-4789-4276-8b9e-c809de6606b6\") " pod="opendatahub/odh-model-controller-769875db56-qprw2" Nov 04 04:53:51.516584 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:51.516559 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa715875-4789-4276-8b9e-c809de6606b6-metrics-certs\") pod \"odh-model-controller-769875db56-qprw2\" (UID: \"fa715875-4789-4276-8b9e-c809de6606b6\") " pod="opendatahub/odh-model-controller-769875db56-qprw2" Nov 04 04:53:51.516775 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:51.516571 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa715875-4789-4276-8b9e-c809de6606b6-cert\") pod \"odh-model-controller-769875db56-qprw2\" (UID: \"fa715875-4789-4276-8b9e-c809de6606b6\") " pod="opendatahub/odh-model-controller-769875db56-qprw2" Nov 04 04:53:51.571417 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:51.571389 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqg2m\" (UniqueName: \"kubernetes.io/projected/fa715875-4789-4276-8b9e-c809de6606b6-kube-api-access-gqg2m\") pod \"odh-model-controller-769875db56-qprw2\" (UID: \"fa715875-4789-4276-8b9e-c809de6606b6\") " pod="opendatahub/odh-model-controller-769875db56-qprw2" Nov 04 04:53:51.662065 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:51.662031 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-qprw2" Nov 04 04:53:52.043240 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:52.043219 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-qprw2"] Nov 04 04:53:52.045791 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:53:52.045757 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa715875_4789_4276_8b9e_c809de6606b6.slice/crio-67cb006d70dd4be5b9ae58023f5bd94821acfdebb77b4d69f0ff9fe13d9f8a3f WatchSource:0}: Error finding container 67cb006d70dd4be5b9ae58023f5bd94821acfdebb77b4d69f0ff9fe13d9f8a3f: Status 404 returned error can't find the container with id 67cb006d70dd4be5b9ae58023f5bd94821acfdebb77b4d69f0ff9fe13d9f8a3f Nov 04 04:53:52.358718 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:52.358682 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:53.014426 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:53.014375 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-qprw2" event={"ID":"fa715875-4789-4276-8b9e-c809de6606b6","Type":"ContainerStarted","Data":"67cb006d70dd4be5b9ae58023f5bd94821acfdebb77b4d69f0ff9fe13d9f8a3f"} Nov 04 04:53:54.849633 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:54.849591 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:55.027599 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:55.027569 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-qprw2" event={"ID":"fa715875-4789-4276-8b9e-c809de6606b6","Type":"ContainerStarted","Data":"575f094ba8b5de89e92f20eae5348ece8e69de92afd81b976dbce57ad3b120a6"} Nov 04 04:53:55.027748 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:55.027687 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-qprw2" Nov 04 04:53:55.143041 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:55.142949 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-qprw2" podStartSLOduration=1.247974254 podStartE2EDuration="4.142932298s" totalImagesPullingTime="2.894958044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:53:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:53:55.142340391 +0000 UTC m=+394.033197414" watchObservedRunningTime="2026-11-04 04:53:55.142932298 +0000 UTC m=+394.033789319" Nov 04 04:53:57.440617 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:57.440579 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:57.560039 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:57.560001 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:57.682907 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:57.682867 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:57.806357 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:57.806273 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:57.931996 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:57.931963 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:58.063672 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:58.063578 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:58.185084 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:58.185057 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:58.314292 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:58.314199 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:58.438456 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:58.438424 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:58.561198 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:58.561161 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:53:58.693261 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:53:58.693220 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:54:02.440659 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:02.440584 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:54:02.579819 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:02.579779 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:54:04.574542 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:04.574506 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:54:04.738886 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:04.738856 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:54:04.861210 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:04.861178 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:54:04.995516 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:04.995484 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:54:05.667919 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:05.667887 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-674d648f7b-hl42s"] Nov 04 04:54:05.671319 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:05.671296 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-674d648f7b-hl42s" Nov 04 04:54:05.774561 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:05.774484 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 04 04:54:05.774561 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:05.774515 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 04:54:05.774561 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:05.774524 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 04:54:05.774561 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:05.774515 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5n9fx\"" Nov 04 04:54:05.776423 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:05.776402 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-674d648f7b-hl42s"] Nov 04 04:54:05.825147 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:05.825110 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 04:54:05.851794 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:05.851764 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44af9590-1593-4b55-9c71-78e5d2b35437-maas-api-tls\") pod \"maas-api-674d648f7b-hl42s\" (UID: \"44af9590-1593-4b55-9c71-78e5d2b35437\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-hl42s" Nov 04 04:54:05.851981 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:05.851801 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44af9590-1593-4b55-9c71-78e5d2b35437-openshift-service-ca\") pod \"maas-api-674d648f7b-hl42s\" (UID: \"44af9590-1593-4b55-9c71-78e5d2b35437\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-hl42s" Nov 04 04:54:05.851981 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:05.851909 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44af9590-1593-4b55-9c71-78e5d2b35437-metrics-certs\") pod \"maas-api-674d648f7b-hl42s\" (UID: \"44af9590-1593-4b55-9c71-78e5d2b35437\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-hl42s" Nov 04 04:54:05.852128 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:05.852014 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgw5p\" (UniqueName: \"kubernetes.io/projected/44af9590-1593-4b55-9c71-78e5d2b35437-kube-api-access-dgw5p\") pod \"maas-api-674d648f7b-hl42s\" (UID: \"44af9590-1593-4b55-9c71-78e5d2b35437\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-hl42s" Nov 04 04:54:05.952839 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:05.952749 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44af9590-1593-4b55-9c71-78e5d2b35437-maas-api-tls\") pod \"maas-api-674d648f7b-hl42s\" (UID: \"44af9590-1593-4b55-9c71-78e5d2b35437\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-hl42s" Nov 04 04:54:05.952839 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:05.952794 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44af9590-1593-4b55-9c71-78e5d2b35437-openshift-service-ca\") pod \"maas-api-674d648f7b-hl42s\" (UID: \"44af9590-1593-4b55-9c71-78e5d2b35437\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-hl42s" Nov 04 04:54:05.952839 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:05.952824 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44af9590-1593-4b55-9c71-78e5d2b35437-metrics-certs\") pod \"maas-api-674d648f7b-hl42s\" (UID: \"44af9590-1593-4b55-9c71-78e5d2b35437\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-hl42s" Nov 04 04:54:05.953139 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:05.952844 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgw5p\" (UniqueName: \"kubernetes.io/projected/44af9590-1593-4b55-9c71-78e5d2b35437-kube-api-access-dgw5p\") pod \"maas-api-674d648f7b-hl42s\" (UID: \"44af9590-1593-4b55-9c71-78e5d2b35437\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-hl42s" Nov 04 04:54:05.953411 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:05.953392 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44af9590-1593-4b55-9c71-78e5d2b35437-openshift-service-ca\") pod \"maas-api-674d648f7b-hl42s\" (UID: \"44af9590-1593-4b55-9c71-78e5d2b35437\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-hl42s" Nov 04 04:54:05.956000 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:05.955959 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44af9590-1593-4b55-9c71-78e5d2b35437-maas-api-tls\") pod \"maas-api-674d648f7b-hl42s\" (UID: \"44af9590-1593-4b55-9c71-78e5d2b35437\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-hl42s" Nov 04 04:54:05.956138 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:05.956121 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44af9590-1593-4b55-9c71-78e5d2b35437-metrics-certs\") pod \"maas-api-674d648f7b-hl42s\" (UID: \"44af9590-1593-4b55-9c71-78e5d2b35437\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-hl42s" Nov 04 04:54:06.017924 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:06.017894 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgw5p\" (UniqueName: \"kubernetes.io/projected/44af9590-1593-4b55-9c71-78e5d2b35437-kube-api-access-dgw5p\") pod \"maas-api-674d648f7b-hl42s\" (UID: \"44af9590-1593-4b55-9c71-78e5d2b35437\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-hl42s" Nov 04 04:54:06.034547 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:06.034519 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-qprw2" Nov 04 04:54:06.284074 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:06.283980 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-674d648f7b-hl42s" Nov 04 04:54:06.656225 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:06.656199 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-674d648f7b-hl42s"] Nov 04 04:54:06.658536 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:54:06.658501 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44af9590_1593_4b55_9c71_78e5d2b35437.slice/crio-4124af5e3b7660b01136576854cf63caea99452e61277bce25e8eec37aa36917 WatchSource:0}: Error finding container 4124af5e3b7660b01136576854cf63caea99452e61277bce25e8eec37aa36917: Status 404 returned error can't find the container with id 4124af5e3b7660b01136576854cf63caea99452e61277bce25e8eec37aa36917 Nov 04 04:54:07.083535 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:07.083448 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-674d648f7b-hl42s" event={"ID":"44af9590-1593-4b55-9c71-78e5d2b35437","Type":"ContainerStarted","Data":"4124af5e3b7660b01136576854cf63caea99452e61277bce25e8eec37aa36917"} Nov 04 04:54:10.106365 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:10.103641 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-674d648f7b-hl42s" event={"ID":"44af9590-1593-4b55-9c71-78e5d2b35437","Type":"ContainerStarted","Data":"7f2705dbf6e2522ad8a52a450c2b9bd5ead2c1a94685559f247c03c67855e443"} Nov 04 04:54:10.106365 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:10.103836 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-674d648f7b-hl42s" Nov 04 04:54:10.220612 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:10.220557 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-674d648f7b-hl42s" podStartSLOduration=2.230000122 podStartE2EDuration="5.220544225s" totalImagesPullingTime="2.990544103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:54:10.219770424 +0000 UTC m=+409.110627481" watchObservedRunningTime="2026-11-04 04:54:10.220544225 +0000 UTC m=+409.111401242" Nov 04 04:54:16.114041 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:16.114015 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-674d648f7b-hl42s" Nov 04 04:54:16.655501 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:16.655456 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:54:16.931388 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:16.931293 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:54:17.183274 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:17.183197 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7f4cf7d6f8-jdnlc"] Nov 04 04:54:17.186725 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:17.186709 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f4cf7d6f8-jdnlc" Nov 04 04:54:17.243414 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:17.243379 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8f86bd-05d2-468a-bdfb-3558700e813c-openshift-service-ca\") pod \"maas-api-7f4cf7d6f8-jdnlc\" (UID: \"4b8f86bd-05d2-468a-bdfb-3558700e813c\") " pod="odh-ai-gateway-infra/maas-api-7f4cf7d6f8-jdnlc" Nov 04 04:54:17.243577 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:17.243463 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tt67\" (UniqueName: \"kubernetes.io/projected/4b8f86bd-05d2-468a-bdfb-3558700e813c-kube-api-access-9tt67\") pod \"maas-api-7f4cf7d6f8-jdnlc\" (UID: \"4b8f86bd-05d2-468a-bdfb-3558700e813c\") " pod="odh-ai-gateway-infra/maas-api-7f4cf7d6f8-jdnlc" Nov 04 04:54:17.243577 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:17.243506 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b8f86bd-05d2-468a-bdfb-3558700e813c-metrics-certs\") pod \"maas-api-7f4cf7d6f8-jdnlc\" (UID: \"4b8f86bd-05d2-468a-bdfb-3558700e813c\") " pod="odh-ai-gateway-infra/maas-api-7f4cf7d6f8-jdnlc" Nov 04 04:54:17.243680 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:17.243595 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8f86bd-05d2-468a-bdfb-3558700e813c-maas-api-tls\") pod \"maas-api-7f4cf7d6f8-jdnlc\" (UID: \"4b8f86bd-05d2-468a-bdfb-3558700e813c\") " pod="odh-ai-gateway-infra/maas-api-7f4cf7d6f8-jdnlc" Nov 04 04:54:17.291381 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:17.291350 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f4cf7d6f8-jdnlc"] Nov 04 04:54:17.344270 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:17.344233 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b8f86bd-05d2-468a-bdfb-3558700e813c-metrics-certs\") pod \"maas-api-7f4cf7d6f8-jdnlc\" (UID: \"4b8f86bd-05d2-468a-bdfb-3558700e813c\") " pod="odh-ai-gateway-infra/maas-api-7f4cf7d6f8-jdnlc" Nov 04 04:54:17.344442 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:17.344276 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8f86bd-05d2-468a-bdfb-3558700e813c-maas-api-tls\") pod \"maas-api-7f4cf7d6f8-jdnlc\" (UID: \"4b8f86bd-05d2-468a-bdfb-3558700e813c\") " pod="odh-ai-gateway-infra/maas-api-7f4cf7d6f8-jdnlc" Nov 04 04:54:17.344442 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:17.344364 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8f86bd-05d2-468a-bdfb-3558700e813c-openshift-service-ca\") pod \"maas-api-7f4cf7d6f8-jdnlc\" (UID: \"4b8f86bd-05d2-468a-bdfb-3558700e813c\") " pod="odh-ai-gateway-infra/maas-api-7f4cf7d6f8-jdnlc" Nov 04 04:54:17.344442 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:17.344438 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tt67\" (UniqueName: \"kubernetes.io/projected/4b8f86bd-05d2-468a-bdfb-3558700e813c-kube-api-access-9tt67\") pod \"maas-api-7f4cf7d6f8-jdnlc\" (UID: \"4b8f86bd-05d2-468a-bdfb-3558700e813c\") " pod="odh-ai-gateway-infra/maas-api-7f4cf7d6f8-jdnlc" Nov 04 04:54:17.345153 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:17.345130 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8f86bd-05d2-468a-bdfb-3558700e813c-openshift-service-ca\") pod \"maas-api-7f4cf7d6f8-jdnlc\" (UID: \"4b8f86bd-05d2-468a-bdfb-3558700e813c\") " pod="odh-ai-gateway-infra/maas-api-7f4cf7d6f8-jdnlc" Nov 04 04:54:17.347170 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:17.347141 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8f86bd-05d2-468a-bdfb-3558700e813c-maas-api-tls\") pod \"maas-api-7f4cf7d6f8-jdnlc\" (UID: \"4b8f86bd-05d2-468a-bdfb-3558700e813c\") " pod="odh-ai-gateway-infra/maas-api-7f4cf7d6f8-jdnlc" Nov 04 04:54:17.347266 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:17.347205 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b8f86bd-05d2-468a-bdfb-3558700e813c-metrics-certs\") pod \"maas-api-7f4cf7d6f8-jdnlc\" (UID: \"4b8f86bd-05d2-468a-bdfb-3558700e813c\") " pod="odh-ai-gateway-infra/maas-api-7f4cf7d6f8-jdnlc" Nov 04 04:54:17.402479 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:17.402443 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tt67\" (UniqueName: \"kubernetes.io/projected/4b8f86bd-05d2-468a-bdfb-3558700e813c-kube-api-access-9tt67\") pod \"maas-api-7f4cf7d6f8-jdnlc\" (UID: \"4b8f86bd-05d2-468a-bdfb-3558700e813c\") " pod="odh-ai-gateway-infra/maas-api-7f4cf7d6f8-jdnlc" Nov 04 04:54:17.499478 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:17.499389 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f4cf7d6f8-jdnlc" Nov 04 04:54:17.872863 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:17.872829 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f4cf7d6f8-jdnlc"] Nov 04 04:54:17.877608 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:54:17.877584 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8f86bd_05d2_468a_bdfb_3558700e813c.slice/crio-10c3a66c783ac09d03ba43910015b1de26406fbae787880e37ff7a440300450d WatchSource:0}: Error finding container 10c3a66c783ac09d03ba43910015b1de26406fbae787880e37ff7a440300450d: Status 404 returned error can't find the container with id 10c3a66c783ac09d03ba43910015b1de26406fbae787880e37ff7a440300450d Nov 04 04:54:17.888158 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:54:17.888129 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 04:54:18.141449 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:18.141368 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f4cf7d6f8-jdnlc" event={"ID":"4b8f86bd-05d2-468a-bdfb-3558700e813c","Type":"ContainerStarted","Data":"10c3a66c783ac09d03ba43910015b1de26406fbae787880e37ff7a440300450d"} Nov 04 04:54:19.147406 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:19.147374 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f4cf7d6f8-jdnlc" event={"ID":"4b8f86bd-05d2-468a-bdfb-3558700e813c","Type":"ContainerStarted","Data":"33e2f912e1415289d6fdc314ebbfc62181c355e8692130d152dd9d164e3f07ed"} Nov 04 04:54:19.147901 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:19.147440 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7f4cf7d6f8-jdnlc" Nov 04 04:54:19.264003 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:19.263931 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7f4cf7d6f8-jdnlc" podStartSLOduration=1.420699221 podStartE2EDuration="2.263916622s" totalImagesPullingTime="843.217401ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:54:19.262929481 +0000 UTC m=+418.153786503" watchObservedRunningTime="2026-11-04 04:54:19.263916622 +0000 UTC m=+418.154773641" Nov 04 04:54:24.042272 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:24.042237 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:54:24.212742 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:24.212707 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:54:25.158267 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:25.158237 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7f4cf7d6f8-jdnlc" Nov 04 04:54:25.325861 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:25.325833 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-674d648f7b-hl42s"] Nov 04 04:54:25.326074 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:25.326054 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-674d648f7b-hl42s" podUID="44af9590-1593-4b55-9c71-78e5d2b35437" containerName="maas-api" containerID="cri-o://7f2705dbf6e2522ad8a52a450c2b9bd5ead2c1a94685559f247c03c67855e443" gracePeriod=30 Nov 04 04:54:25.570109 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:25.570085 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-674d648f7b-hl42s" Nov 04 04:54:25.617087 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:25.617061 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44af9590-1593-4b55-9c71-78e5d2b35437-metrics-certs\" (UniqueName: \"kubernetes.io/secret/44af9590-1593-4b55-9c71-78e5d2b35437-metrics-certs\") pod \"44af9590-1593-4b55-9c71-78e5d2b35437\" (UID: \"44af9590-1593-4b55-9c71-78e5d2b35437\") " Nov 04 04:54:25.617203 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:25.617113 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44af9590-1593-4b55-9c71-78e5d2b35437-kube-api-access-dgw5p\" (UniqueName: \"kubernetes.io/projected/44af9590-1593-4b55-9c71-78e5d2b35437-kube-api-access-dgw5p\") pod \"44af9590-1593-4b55-9c71-78e5d2b35437\" (UID: \"44af9590-1593-4b55-9c71-78e5d2b35437\") " Nov 04 04:54:25.617203 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:25.617165 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44af9590-1593-4b55-9c71-78e5d2b35437-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44af9590-1593-4b55-9c71-78e5d2b35437-openshift-service-ca\") pod \"44af9590-1593-4b55-9c71-78e5d2b35437\" (UID: \"44af9590-1593-4b55-9c71-78e5d2b35437\") " Nov 04 04:54:25.617295 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:25.617218 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44af9590-1593-4b55-9c71-78e5d2b35437-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44af9590-1593-4b55-9c71-78e5d2b35437-maas-api-tls\") pod \"44af9590-1593-4b55-9c71-78e5d2b35437\" (UID: \"44af9590-1593-4b55-9c71-78e5d2b35437\") " Nov 04 04:54:25.617577 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:25.617553 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44af9590-1593-4b55-9c71-78e5d2b35437-openshift-service-ca" pod "44af9590-1593-4b55-9c71-78e5d2b35437" (UID: "44af9590-1593-4b55-9c71-78e5d2b35437"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:54:25.619409 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:25.619379 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44af9590-1593-4b55-9c71-78e5d2b35437-kube-api-access-dgw5p" pod "44af9590-1593-4b55-9c71-78e5d2b35437" (UID: "44af9590-1593-4b55-9c71-78e5d2b35437"). InnerVolumeSpecName "kube-api-access-dgw5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:54:25.619491 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:25.619400 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44af9590-1593-4b55-9c71-78e5d2b35437-maas-api-tls" pod "44af9590-1593-4b55-9c71-78e5d2b35437" (UID: "44af9590-1593-4b55-9c71-78e5d2b35437"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:54:25.619491 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:25.619423 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44af9590-1593-4b55-9c71-78e5d2b35437-metrics-certs" pod "44af9590-1593-4b55-9c71-78e5d2b35437" (UID: "44af9590-1593-4b55-9c71-78e5d2b35437"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:54:25.718040 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:25.717928 4581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44af9590-1593-4b55-9c71-78e5d2b35437-metrics-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:54:25.718040 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:25.717960 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgw5p\" (UniqueName: \"kubernetes.io/projected/44af9590-1593-4b55-9c71-78e5d2b35437-kube-api-access-dgw5p\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:54:25.718040 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:25.717975 4581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44af9590-1593-4b55-9c71-78e5d2b35437-openshift-service-ca\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:54:25.718040 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:25.718014 4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44af9590-1593-4b55-9c71-78e5d2b35437-maas-api-tls\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:54:26.181364 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:26.181332 4581 generic.go:356] "Generic (PLEG): container finished" podID="44af9590-1593-4b55-9c71-78e5d2b35437" containerID="7f2705dbf6e2522ad8a52a450c2b9bd5ead2c1a94685559f247c03c67855e443" exitCode=0 Nov 04 04:54:26.181747 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:26.181377 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-674d648f7b-hl42s" Nov 04 04:54:26.181747 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:26.181420 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-674d648f7b-hl42s" event={"ID":"44af9590-1593-4b55-9c71-78e5d2b35437","Type":"ContainerDied","Data":"7f2705dbf6e2522ad8a52a450c2b9bd5ead2c1a94685559f247c03c67855e443"} Nov 04 04:54:26.181747 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:26.181475 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-674d648f7b-hl42s" event={"ID":"44af9590-1593-4b55-9c71-78e5d2b35437","Type":"ContainerDied","Data":"4124af5e3b7660b01136576854cf63caea99452e61277bce25e8eec37aa36917"} Nov 04 04:54:26.181747 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:26.181498 4581 scope.go:122] "RemoveContainer" containerID="7f2705dbf6e2522ad8a52a450c2b9bd5ead2c1a94685559f247c03c67855e443" Nov 04 04:54:26.191125 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:26.191110 4581 scope.go:122] "RemoveContainer" containerID="7f2705dbf6e2522ad8a52a450c2b9bd5ead2c1a94685559f247c03c67855e443" Nov 04 04:54:26.191419 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:54:26.191402 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f2705dbf6e2522ad8a52a450c2b9bd5ead2c1a94685559f247c03c67855e443\": container with ID starting with 7f2705dbf6e2522ad8a52a450c2b9bd5ead2c1a94685559f247c03c67855e443 not found: ID does not exist" containerID="7f2705dbf6e2522ad8a52a450c2b9bd5ead2c1a94685559f247c03c67855e443" Nov 04 04:54:26.191479 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:26.191429 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f2705dbf6e2522ad8a52a450c2b9bd5ead2c1a94685559f247c03c67855e443"} err="failed to get container status \"7f2705dbf6e2522ad8a52a450c2b9bd5ead2c1a94685559f247c03c67855e443\": rpc error: code = NotFound desc = could not find container \"7f2705dbf6e2522ad8a52a450c2b9bd5ead2c1a94685559f247c03c67855e443\": container with ID starting with 7f2705dbf6e2522ad8a52a450c2b9bd5ead2c1a94685559f247c03c67855e443 not found: ID does not exist" Nov 04 04:54:26.346948 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:26.346917 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-674d648f7b-hl42s"] Nov 04 04:54:26.350378 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:26.350353 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-674d648f7b-hl42s"] Nov 04 04:54:27.230518 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:27.230485 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44af9590-1593-4b55-9c71-78e5d2b35437" path="/var/lib/kubelet/pods/44af9590-1593-4b55-9c71-78e5d2b35437/volumes" Nov 04 04:54:47.886245 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:54:47.886203 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 04:54:58.707640 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:54:58.707606 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:55:15.102034 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:55:15.101973 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:55:17.885744 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:55:17.885702 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 04:55:36.752589 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:55:36.752512 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:55:36.885027 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:55:36.884967 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:55:47.885130 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:55:47.885094 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 04:55:57.645214 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:55:57.645183 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:55:59.033197 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:55:59.033157 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:56:02.361854 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.361820 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f"] Nov 04 04:56:02.362271 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.362185 4581 cpu_manager.go:414] "containerMap: removing container" podUID="44af9590-1593-4b55-9c71-78e5d2b35437" containerName="maas-api" Nov 04 04:56:02.362271 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.362198 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44af9590-1593-4b55-9c71-78e5d2b35437" containerName="maas-api" Nov 04 04:56:02.362350 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.362278 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="44af9590-1593-4b55-9c71-78e5d2b35437" containerName="maas-api" Nov 04 04:56:02.365330 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.365314 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" Nov 04 04:56:02.445605 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.445568 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:56:02.463643 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.463607 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca9c82d3-d38a-4910-8a3b-90c4412b508b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f\" (UID: \"ca9c82d3-d38a-4910-8a3b-90c4412b508b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" Nov 04 04:56:02.463780 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.463662 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca9c82d3-d38a-4910-8a3b-90c4412b508b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f\" (UID: \"ca9c82d3-d38a-4910-8a3b-90c4412b508b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" Nov 04 04:56:02.463780 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.463732 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca9c82d3-d38a-4910-8a3b-90c4412b508b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f\" (UID: \"ca9c82d3-d38a-4910-8a3b-90c4412b508b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" Nov 04 04:56:02.463780 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.463749 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca9c82d3-d38a-4910-8a3b-90c4412b508b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f\" (UID: \"ca9c82d3-d38a-4910-8a3b-90c4412b508b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" Nov 04 04:56:02.463880 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.463810 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz4gq\" (UniqueName: \"kubernetes.io/projected/ca9c82d3-d38a-4910-8a3b-90c4412b508b-kube-api-access-cz4gq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f\" (UID: \"ca9c82d3-d38a-4910-8a3b-90c4412b508b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" Nov 04 04:56:02.463916 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.463882 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca9c82d3-d38a-4910-8a3b-90c4412b508b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f\" (UID: \"ca9c82d3-d38a-4910-8a3b-90c4412b508b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" Nov 04 04:56:02.468682 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.468665 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 04 04:56:02.468743 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.468702 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 04 04:56:02.468840 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.468829 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wx27q\"" Nov 04 04:56:02.468894 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.468885 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 04 04:56:02.470475 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.470458 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f"] Nov 04 04:56:02.565163 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.565128 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca9c82d3-d38a-4910-8a3b-90c4412b508b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f\" (UID: \"ca9c82d3-d38a-4910-8a3b-90c4412b508b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" Nov 04 04:56:02.565163 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.565170 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca9c82d3-d38a-4910-8a3b-90c4412b508b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f\" (UID: \"ca9c82d3-d38a-4910-8a3b-90c4412b508b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" Nov 04 04:56:02.565427 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.565316 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca9c82d3-d38a-4910-8a3b-90c4412b508b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f\" (UID: \"ca9c82d3-d38a-4910-8a3b-90c4412b508b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" Nov 04 04:56:02.565427 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.565378 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca9c82d3-d38a-4910-8a3b-90c4412b508b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f\" (UID: \"ca9c82d3-d38a-4910-8a3b-90c4412b508b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" Nov 04 04:56:02.565427 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.565405 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca9c82d3-d38a-4910-8a3b-90c4412b508b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f\" (UID: \"ca9c82d3-d38a-4910-8a3b-90c4412b508b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" Nov 04 04:56:02.565590 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.565434 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz4gq\" (UniqueName: \"kubernetes.io/projected/ca9c82d3-d38a-4910-8a3b-90c4412b508b-kube-api-access-cz4gq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f\" (UID: \"ca9c82d3-d38a-4910-8a3b-90c4412b508b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" Nov 04 04:56:02.565590 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.565558 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca9c82d3-d38a-4910-8a3b-90c4412b508b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f\" (UID: \"ca9c82d3-d38a-4910-8a3b-90c4412b508b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" Nov 04 04:56:02.565676 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.565632 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca9c82d3-d38a-4910-8a3b-90c4412b508b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f\" (UID: \"ca9c82d3-d38a-4910-8a3b-90c4412b508b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" Nov 04 04:56:02.565741 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.565720 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca9c82d3-d38a-4910-8a3b-90c4412b508b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f\" (UID: \"ca9c82d3-d38a-4910-8a3b-90c4412b508b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" Nov 04 04:56:02.567665 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.567643 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca9c82d3-d38a-4910-8a3b-90c4412b508b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f\" (UID: \"ca9c82d3-d38a-4910-8a3b-90c4412b508b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" Nov 04 04:56:02.567981 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.567962 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca9c82d3-d38a-4910-8a3b-90c4412b508b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f\" (UID: \"ca9c82d3-d38a-4910-8a3b-90c4412b508b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" Nov 04 04:56:02.626832 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.626753 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz4gq\" (UniqueName: \"kubernetes.io/projected/ca9c82d3-d38a-4910-8a3b-90c4412b508b-kube-api-access-cz4gq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f\" (UID: \"ca9c82d3-d38a-4910-8a3b-90c4412b508b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" Nov 04 04:56:02.678053 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:02.678016 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" Nov 04 04:56:03.049999 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:03.049958 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f"] Nov 04 04:56:03.052298 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:56:03.052276 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9c82d3_d38a_4910_8a3b_90c4412b508b.slice/crio-1aa644e2a0d7eb852ecd54cb1fdac4d9c22df21fe469bc67dc7ad64d9cc2fe29 WatchSource:0}: Error finding container 1aa644e2a0d7eb852ecd54cb1fdac4d9c22df21fe469bc67dc7ad64d9cc2fe29: Status 404 returned error can't find the container with id 1aa644e2a0d7eb852ecd54cb1fdac4d9c22df21fe469bc67dc7ad64d9cc2fe29 Nov 04 04:56:03.645639 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:03.645588 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" event={"ID":"ca9c82d3-d38a-4910-8a3b-90c4412b508b","Type":"ContainerStarted","Data":"1aa644e2a0d7eb852ecd54cb1fdac4d9c22df21fe469bc67dc7ad64d9cc2fe29"} Nov 04 04:56:04.651996 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:04.651954 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" event={"ID":"ca9c82d3-d38a-4910-8a3b-90c4412b508b","Type":"ContainerStarted","Data":"50cd4315ab67b94c40545514d69789adba4e877c10c2b156067d5694df51e716"} Nov 04 04:56:04.678756 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:04.678727 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" Nov 04 04:56:04.768135 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:04.768084 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" podStartSLOduration=1.255340546 podStartE2EDuration="2.768069016s" totalImagesPullingTime="1.51272847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:56:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:56:04.766467466 +0000 UTC m=+523.657324487" watchObservedRunningTime="2026-11-04 04:56:04.768069016 +0000 UTC m=+523.658926036" Nov 04 04:56:05.326888 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:05.326852 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-kp5cw"] Nov 04 04:56:05.330193 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:05.330177 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" Nov 04 04:56:05.388228 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:05.388196 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-kp5cw"] Nov 04 04:56:05.398305 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:05.398270 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26f5c90c-d608-4b86-9909-5c1bf8fdbd24-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-kp5cw\" (UID: \"26f5c90c-d608-4b86-9909-5c1bf8fdbd24\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" Nov 04 04:56:05.398488 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:05.398339 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26f5c90c-d608-4b86-9909-5c1bf8fdbd24-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-kp5cw\" (UID: \"26f5c90c-d608-4b86-9909-5c1bf8fdbd24\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" Nov 04 04:56:05.398547 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:05.398489 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26f5c90c-d608-4b86-9909-5c1bf8fdbd24-home\") pod \"echo-simulator-kserve-64b6f8f96c-kp5cw\" (UID: \"26f5c90c-d608-4b86-9909-5c1bf8fdbd24\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" Nov 04 04:56:05.398599 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:05.398551 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f5c90c-d608-4b86-9909-5c1bf8fdbd24-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-kp5cw\" (UID: \"26f5c90c-d608-4b86-9909-5c1bf8fdbd24\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" Nov 04 04:56:05.398666 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:05.398635 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26f5c90c-d608-4b86-9909-5c1bf8fdbd24-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-kp5cw\" (UID: \"26f5c90c-d608-4b86-9909-5c1bf8fdbd24\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" Nov 04 04:56:05.398706 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:05.398686 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6rfr\" (UniqueName: \"kubernetes.io/projected/26f5c90c-d608-4b86-9909-5c1bf8fdbd24-kube-api-access-h6rfr\") pod \"echo-simulator-kserve-64b6f8f96c-kp5cw\" (UID: \"26f5c90c-d608-4b86-9909-5c1bf8fdbd24\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" Nov 04 04:56:05.405789 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:05.405751 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:56:05.435822 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:05.435796 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 04 04:56:05.499168 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:05.499127 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26f5c90c-d608-4b86-9909-5c1bf8fdbd24-home\") pod \"echo-simulator-kserve-64b6f8f96c-kp5cw\" (UID: \"26f5c90c-d608-4b86-9909-5c1bf8fdbd24\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" Nov 04 04:56:05.499362 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:05.499202 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f5c90c-d608-4b86-9909-5c1bf8fdbd24-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-kp5cw\" (UID: \"26f5c90c-d608-4b86-9909-5c1bf8fdbd24\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" Nov 04 04:56:05.499362 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:05.499242 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26f5c90c-d608-4b86-9909-5c1bf8fdbd24-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-kp5cw\" (UID: \"26f5c90c-d608-4b86-9909-5c1bf8fdbd24\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" Nov 04 04:56:05.499362 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:05.499266 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6rfr\" (UniqueName: \"kubernetes.io/projected/26f5c90c-d608-4b86-9909-5c1bf8fdbd24-kube-api-access-h6rfr\") pod \"echo-simulator-kserve-64b6f8f96c-kp5cw\" (UID: \"26f5c90c-d608-4b86-9909-5c1bf8fdbd24\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" Nov 04 04:56:05.499362 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:05.499313 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26f5c90c-d608-4b86-9909-5c1bf8fdbd24-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-kp5cw\" (UID: \"26f5c90c-d608-4b86-9909-5c1bf8fdbd24\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" Nov 04 04:56:05.499555 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:05.499364 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26f5c90c-d608-4b86-9909-5c1bf8fdbd24-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-kp5cw\" (UID: \"26f5c90c-d608-4b86-9909-5c1bf8fdbd24\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" Nov 04 04:56:05.499609 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:05.499578 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26f5c90c-d608-4b86-9909-5c1bf8fdbd24-home\") pod \"echo-simulator-kserve-64b6f8f96c-kp5cw\" (UID: \"26f5c90c-d608-4b86-9909-5c1bf8fdbd24\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" Nov 04 04:56:05.499687 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:05.499607 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f5c90c-d608-4b86-9909-5c1bf8fdbd24-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-kp5cw\" (UID: \"26f5c90c-d608-4b86-9909-5c1bf8fdbd24\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" Nov 04 04:56:05.499747 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:05.499734 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26f5c90c-d608-4b86-9909-5c1bf8fdbd24-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-kp5cw\" (UID: \"26f5c90c-d608-4b86-9909-5c1bf8fdbd24\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" Nov 04 04:56:05.501853 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:05.501831 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26f5c90c-d608-4b86-9909-5c1bf8fdbd24-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-kp5cw\" (UID: \"26f5c90c-d608-4b86-9909-5c1bf8fdbd24\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" Nov 04 04:56:05.502086 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:05.502068 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26f5c90c-d608-4b86-9909-5c1bf8fdbd24-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-kp5cw\" (UID: \"26f5c90c-d608-4b86-9909-5c1bf8fdbd24\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" Nov 04 04:56:05.557244 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:05.557214 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6rfr\" (UniqueName: \"kubernetes.io/projected/26f5c90c-d608-4b86-9909-5c1bf8fdbd24-kube-api-access-h6rfr\") pod \"echo-simulator-kserve-64b6f8f96c-kp5cw\" (UID: \"26f5c90c-d608-4b86-9909-5c1bf8fdbd24\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" Nov 04 04:56:05.642973 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:05.642924 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" Nov 04 04:56:06.017394 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:06.017369 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-kp5cw"] Nov 04 04:56:06.019815 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:56:06.019781 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f5c90c_d608_4b86_9909_5c1bf8fdbd24.slice/crio-7c5cc3ad7d7a5abd58f3e16400d7353e5b6677260c3b10229093871e8f4d08e9 WatchSource:0}: Error finding container 7c5cc3ad7d7a5abd58f3e16400d7353e5b6677260c3b10229093871e8f4d08e9: Status 404 returned error can't find the container with id 7c5cc3ad7d7a5abd58f3e16400d7353e5b6677260c3b10229093871e8f4d08e9 Nov 04 04:56:06.111735 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:06.111697 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:56:06.236350 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:06.236319 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:56:06.664541 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:06.664506 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" event={"ID":"26f5c90c-d608-4b86-9909-5c1bf8fdbd24","Type":"ContainerStarted","Data":"7c5cc3ad7d7a5abd58f3e16400d7353e5b6677260c3b10229093871e8f4d08e9"} Nov 04 04:56:08.523774 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:08.523744 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:56:09.682403 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:09.682367 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" event={"ID":"26f5c90c-d608-4b86-9909-5c1bf8fdbd24","Type":"ContainerStarted","Data":"0efab478dd939e3ed8882889ff860dbbc30096d85b51905a8bc004e812837506"} Nov 04 04:56:09.797700 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:09.797637 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" podStartSLOduration=2.214052099 podStartE2EDuration="4.797623729s" totalImagesPullingTime="2.58357163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:56:09.796348814 +0000 UTC m=+528.687205835" watchObservedRunningTime="2026-11-04 04:56:09.797623729 +0000 UTC m=+528.688480745" Nov 04 04:56:10.644086 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:10.644056 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" Nov 04 04:56:12.522253 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:12.522223 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:56:12.678822 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:12.678789 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" Nov 04 04:56:12.691426 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:12.691398 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" Nov 04 04:56:12.707611 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:12.707585 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f" Nov 04 04:56:15.643934 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:15.643899 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" Nov 04 04:56:15.656421 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:15.656399 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" Nov 04 04:56:15.721791 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:15.721751 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-kp5cw" Nov 04 04:56:15.923895 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:15.923867 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h"] Nov 04 04:56:15.927490 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:15.927467 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" Nov 04 04:56:15.980286 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:15.980252 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 04 04:56:15.999603 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:15.999576 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae7aa7c9-fba5-4b19-9163-32de4ad7b567-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v454h\" (UID: \"ae7aa7c9-fba5-4b19-9163-32de4ad7b567\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" Nov 04 04:56:15.999723 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:15.999696 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae7aa7c9-fba5-4b19-9163-32de4ad7b567-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v454h\" (UID: \"ae7aa7c9-fba5-4b19-9163-32de4ad7b567\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" Nov 04 04:56:15.999767 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:15.999718 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae7aa7c9-fba5-4b19-9163-32de4ad7b567-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v454h\" (UID: \"ae7aa7c9-fba5-4b19-9163-32de4ad7b567\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" Nov 04 04:56:15.999814 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:15.999766 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhrcq\" (UniqueName: \"kubernetes.io/projected/ae7aa7c9-fba5-4b19-9163-32de4ad7b567-kube-api-access-nhrcq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v454h\" (UID: \"ae7aa7c9-fba5-4b19-9163-32de4ad7b567\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" Nov 04 04:56:15.999814 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:15.999795 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae7aa7c9-fba5-4b19-9163-32de4ad7b567-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v454h\" (UID: \"ae7aa7c9-fba5-4b19-9163-32de4ad7b567\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" Nov 04 04:56:15.999917 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:15.999876 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae7aa7c9-fba5-4b19-9163-32de4ad7b567-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v454h\" (UID: \"ae7aa7c9-fba5-4b19-9163-32de4ad7b567\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" Nov 04 04:56:16.003202 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:16.003181 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h"] Nov 04 04:56:16.007587 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:16.007562 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:56:16.100756 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:16.100730 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae7aa7c9-fba5-4b19-9163-32de4ad7b567-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v454h\" (UID: \"ae7aa7c9-fba5-4b19-9163-32de4ad7b567\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" Nov 04 04:56:16.100924 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:16.100761 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae7aa7c9-fba5-4b19-9163-32de4ad7b567-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v454h\" (UID: \"ae7aa7c9-fba5-4b19-9163-32de4ad7b567\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" Nov 04 04:56:16.100924 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:16.100795 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhrcq\" (UniqueName: \"kubernetes.io/projected/ae7aa7c9-fba5-4b19-9163-32de4ad7b567-kube-api-access-nhrcq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v454h\" (UID: \"ae7aa7c9-fba5-4b19-9163-32de4ad7b567\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" Nov 04 04:56:16.100924 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:16.100821 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae7aa7c9-fba5-4b19-9163-32de4ad7b567-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v454h\" (UID: \"ae7aa7c9-fba5-4b19-9163-32de4ad7b567\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" Nov 04 04:56:16.101144 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:16.100938 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae7aa7c9-fba5-4b19-9163-32de4ad7b567-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v454h\" (UID: \"ae7aa7c9-fba5-4b19-9163-32de4ad7b567\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" Nov 04 04:56:16.101144 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:16.101059 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae7aa7c9-fba5-4b19-9163-32de4ad7b567-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v454h\" (UID: \"ae7aa7c9-fba5-4b19-9163-32de4ad7b567\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" Nov 04 04:56:16.101240 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:16.101195 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae7aa7c9-fba5-4b19-9163-32de4ad7b567-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v454h\" (UID: \"ae7aa7c9-fba5-4b19-9163-32de4ad7b567\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" Nov 04 04:56:16.101333 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:16.101310 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae7aa7c9-fba5-4b19-9163-32de4ad7b567-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v454h\" (UID: \"ae7aa7c9-fba5-4b19-9163-32de4ad7b567\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" Nov 04 04:56:16.101397 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:16.101375 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae7aa7c9-fba5-4b19-9163-32de4ad7b567-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v454h\" (UID: \"ae7aa7c9-fba5-4b19-9163-32de4ad7b567\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" Nov 04 04:56:16.103256 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:16.103238 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae7aa7c9-fba5-4b19-9163-32de4ad7b567-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v454h\" (UID: \"ae7aa7c9-fba5-4b19-9163-32de4ad7b567\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" Nov 04 04:56:16.103415 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:16.103399 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae7aa7c9-fba5-4b19-9163-32de4ad7b567-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v454h\" (UID: \"ae7aa7c9-fba5-4b19-9163-32de4ad7b567\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" Nov 04 04:56:16.160671 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:16.160640 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhrcq\" (UniqueName: \"kubernetes.io/projected/ae7aa7c9-fba5-4b19-9163-32de4ad7b567-kube-api-access-nhrcq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v454h\" (UID: \"ae7aa7c9-fba5-4b19-9163-32de4ad7b567\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" Nov 04 04:56:16.240104 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:16.240028 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" Nov 04 04:56:16.611698 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:16.611592 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h"] Nov 04 04:56:16.613919 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:56:16.613891 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7aa7c9_fba5_4b19_9163_32de4ad7b567.slice/crio-42d079c00989fc6138e829bbba534529d9f104125937f0ca5787e4eed106e323 WatchSource:0}: Error finding container 42d079c00989fc6138e829bbba534529d9f104125937f0ca5787e4eed106e323: Status 404 returned error can't find the container with id 42d079c00989fc6138e829bbba534529d9f104125937f0ca5787e4eed106e323 Nov 04 04:56:16.716413 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:16.716383 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" event={"ID":"ae7aa7c9-fba5-4b19-9163-32de4ad7b567","Type":"ContainerStarted","Data":"42d079c00989fc6138e829bbba534529d9f104125937f0ca5787e4eed106e323"} Nov 04 04:56:17.723099 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:17.723070 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" event={"ID":"ae7aa7c9-fba5-4b19-9163-32de4ad7b567","Type":"ContainerStarted","Data":"318f73e47e7a57121cd667eeb96d5de72ff1e35888c1528b443365cd0d204f20"} Nov 04 04:56:17.839492 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:17.839435 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" podStartSLOduration=2.250692523 podStartE2EDuration="2.839421279s" totalImagesPullingTime="588.728756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:56:17.838289504 +0000 UTC m=+536.729146525" watchObservedRunningTime="2026-11-04 04:56:17.839421279 +0000 UTC m=+536.730278298" Nov 04 04:56:17.885462 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:56:17.885428 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 04:56:18.241214 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:18.241189 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" Nov 04 04:56:23.042764 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:23.042727 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:56:26.240869 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.240838 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" Nov 04 04:56:26.253554 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.253532 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" Nov 04 04:56:26.360904 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.360870 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r"] Nov 04 04:56:26.367158 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.367138 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" Nov 04 04:56:26.422977 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.422941 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r"] Nov 04 04:56:26.423183 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.423169 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 04 04:56:26.443766 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.443736 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:56:26.493083 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.493009 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c20fd01-1367-4eef-be7c-acb7a33be306-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mmz8r\" (UID: \"1c20fd01-1367-4eef-be7c-acb7a33be306\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" Nov 04 04:56:26.493083 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.493049 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c20fd01-1367-4eef-be7c-acb7a33be306-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mmz8r\" (UID: \"1c20fd01-1367-4eef-be7c-acb7a33be306\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" Nov 04 04:56:26.493293 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.493142 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c20fd01-1367-4eef-be7c-acb7a33be306-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mmz8r\" (UID: \"1c20fd01-1367-4eef-be7c-acb7a33be306\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" Nov 04 04:56:26.493293 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.493193 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c20fd01-1367-4eef-be7c-acb7a33be306-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mmz8r\" (UID: \"1c20fd01-1367-4eef-be7c-acb7a33be306\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" Nov 04 04:56:26.493293 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.493210 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c20fd01-1367-4eef-be7c-acb7a33be306-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mmz8r\" (UID: \"1c20fd01-1367-4eef-be7c-acb7a33be306\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" Nov 04 04:56:26.493293 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.493290 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kxpl\" (UniqueName: \"kubernetes.io/projected/1c20fd01-1367-4eef-be7c-acb7a33be306-kube-api-access-6kxpl\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mmz8r\" (UID: \"1c20fd01-1367-4eef-be7c-acb7a33be306\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" Nov 04 04:56:26.594073 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.594039 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c20fd01-1367-4eef-be7c-acb7a33be306-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mmz8r\" (UID: \"1c20fd01-1367-4eef-be7c-acb7a33be306\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" Nov 04 04:56:26.594073 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.594077 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c20fd01-1367-4eef-be7c-acb7a33be306-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mmz8r\" (UID: \"1c20fd01-1367-4eef-be7c-acb7a33be306\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" Nov 04 04:56:26.594282 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.594112 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c20fd01-1367-4eef-be7c-acb7a33be306-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mmz8r\" (UID: \"1c20fd01-1367-4eef-be7c-acb7a33be306\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" Nov 04 04:56:26.594282 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.594155 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c20fd01-1367-4eef-be7c-acb7a33be306-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mmz8r\" (UID: \"1c20fd01-1367-4eef-be7c-acb7a33be306\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" Nov 04 04:56:26.594282 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.594178 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c20fd01-1367-4eef-be7c-acb7a33be306-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mmz8r\" (UID: \"1c20fd01-1367-4eef-be7c-acb7a33be306\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" Nov 04 04:56:26.594282 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.594207 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kxpl\" (UniqueName: \"kubernetes.io/projected/1c20fd01-1367-4eef-be7c-acb7a33be306-kube-api-access-6kxpl\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mmz8r\" (UID: \"1c20fd01-1367-4eef-be7c-acb7a33be306\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" Nov 04 04:56:26.594514 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.594494 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c20fd01-1367-4eef-be7c-acb7a33be306-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mmz8r\" (UID: \"1c20fd01-1367-4eef-be7c-acb7a33be306\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" Nov 04 04:56:26.594596 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.594525 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c20fd01-1367-4eef-be7c-acb7a33be306-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mmz8r\" (UID: \"1c20fd01-1367-4eef-be7c-acb7a33be306\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" Nov 04 04:56:26.594639 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.594601 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c20fd01-1367-4eef-be7c-acb7a33be306-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mmz8r\" (UID: \"1c20fd01-1367-4eef-be7c-acb7a33be306\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" Nov 04 04:56:26.596519 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.596500 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c20fd01-1367-4eef-be7c-acb7a33be306-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mmz8r\" (UID: \"1c20fd01-1367-4eef-be7c-acb7a33be306\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" Nov 04 04:56:26.596939 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.596925 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c20fd01-1367-4eef-be7c-acb7a33be306-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mmz8r\" (UID: \"1c20fd01-1367-4eef-be7c-acb7a33be306\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" Nov 04 04:56:26.653798 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.653761 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kxpl\" (UniqueName: \"kubernetes.io/projected/1c20fd01-1367-4eef-be7c-acb7a33be306-kube-api-access-6kxpl\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mmz8r\" (UID: \"1c20fd01-1367-4eef-be7c-acb7a33be306\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" Nov 04 04:56:26.680662 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.680638 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" Nov 04 04:56:26.784418 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:26.784342 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v454h" Nov 04 04:56:27.055579 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:27.055491 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r"] Nov 04 04:56:27.058347 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:56:27.058317 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c20fd01_1367_4eef_be7c_acb7a33be306.slice/crio-14dd4b39ea4ec18929b1265cf4e3c5b7b2606a1eeb372bb4cd2eb13262b740ff WatchSource:0}: Error finding container 14dd4b39ea4ec18929b1265cf4e3c5b7b2606a1eeb372bb4cd2eb13262b740ff: Status 404 returned error can't find the container with id 14dd4b39ea4ec18929b1265cf4e3c5b7b2606a1eeb372bb4cd2eb13262b740ff Nov 04 04:56:27.779425 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:27.779388 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" event={"ID":"1c20fd01-1367-4eef-be7c-acb7a33be306","Type":"ContainerStarted","Data":"a105b35d2f2f74ccc4eb4ee216f545adb0c037a8b013ceaaebf29ecef2db35b7"} Nov 04 04:56:27.779793 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:27.779448 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" event={"ID":"1c20fd01-1367-4eef-be7c-acb7a33be306","Type":"ContainerStarted","Data":"14dd4b39ea4ec18929b1265cf4e3c5b7b2606a1eeb372bb4cd2eb13262b740ff"} Nov 04 04:56:27.896333 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:27.896235 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" podStartSLOduration=1.327754006 podStartE2EDuration="1.896220155s" totalImagesPullingTime="568.466149ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:56:27.895140659 +0000 UTC m=+546.785997679" watchObservedRunningTime="2026-11-04 04:56:27.896220155 +0000 UTC m=+546.787077177" Nov 04 04:56:28.681183 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:28.681149 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" Nov 04 04:56:30.407681 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.407652 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc"] Nov 04 04:56:30.415209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.415187 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.467724 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.467699 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-p9hzm\"" Nov 04 04:56:30.518140 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.518105 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 04 04:56:30.530779 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.530740 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.530950 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.530819 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.530950 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.530856 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.530950 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.530926 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.531151 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.530964 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.531151 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.531023 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a91d185a-8a3f-4a08-9395-72e9a92265c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.531151 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.531062 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.531151 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.531104 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.531151 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.531142 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwdsx\" (UniqueName: \"kubernetes.io/projected/a91d185a-8a3f-4a08-9395-72e9a92265c8-kube-api-access-mwdsx\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.619954 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.619915 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc"] Nov 04 04:56:30.632442 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.632389 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.632442 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.632439 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a91d185a-8a3f-4a08-9395-72e9a92265c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.632830 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.632476 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.632830 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.632512 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.632830 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.632543 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwdsx\" (UniqueName: \"kubernetes.io/projected/a91d185a-8a3f-4a08-9395-72e9a92265c8-kube-api-access-mwdsx\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.632830 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.632631 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.632830 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.632665 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.632830 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.632680 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.632830 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.632740 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.633228 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.632868 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.633393 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.633364 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.633486 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.633434 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.633566 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.633439 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.633951 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.633934 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a91d185a-8a3f-4a08-9395-72e9a92265c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.635140 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.635124 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.635913 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.635887 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.690788 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.690721 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.690907 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.690888 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwdsx\" (UniqueName: \"kubernetes.io/projected/a91d185a-8a3f-4a08-9395-72e9a92265c8-kube-api-access-mwdsx\") pod \"maas-default-gateway-openshift-default-6886b96f9-bnxlc\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:30.727965 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:30.727937 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:31.206213 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:31.206185 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc"] Nov 04 04:56:31.208041 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:56:31.207974 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda91d185a_8a3f_4a08_9395_72e9a92265c8.slice/crio-57d46108f35865374dc571ec6f4a7523da33952e78f816e2a0a223294139b327 WatchSource:0}: Error finding container 57d46108f35865374dc571ec6f4a7523da33952e78f816e2a0a223294139b327: Status 404 returned error can't find the container with id 57d46108f35865374dc571ec6f4a7523da33952e78f816e2a0a223294139b327 Nov 04 04:56:31.805173 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:31.805134 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" event={"ID":"a91d185a-8a3f-4a08-9395-72e9a92265c8","Type":"ContainerStarted","Data":"57d46108f35865374dc571ec6f4a7523da33952e78f816e2a0a223294139b327"} Nov 04 04:56:35.017639 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:35.017606 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:56:35.017958 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:35.017674 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:56:35.017958 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:35.017706 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:56:35.828861 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:35.828825 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" event={"ID":"a91d185a-8a3f-4a08-9395-72e9a92265c8","Type":"ContainerStarted","Data":"9a3c03eda84cdce5dff28957661ad15e6de30431da0dddcb178d9c22878b3bff"} Nov 04 04:56:36.051622 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:36.051565 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" podStartSLOduration=2.244956778 podStartE2EDuration="6.051551542s" totalImagesPullingTime="3.806594764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:56:36.000661855 +0000 UTC m=+554.891518904" watchObservedRunningTime="2026-11-04 04:56:36.051551542 +0000 UTC m=+554.942408561" Nov 04 04:56:36.681358 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:36.681321 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" Nov 04 04:56:36.693819 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:36.693790 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" Nov 04 04:56:36.728311 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:36.728290 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:36.729474 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:36.729451 4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bnxlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 04 04:56:36.729608 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:36.729511 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" podUID="a91d185a-8a3f-4a08-9395-72e9a92265c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 04 04:56:36.844289 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:36.844265 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mmz8r" Nov 04 04:56:37.729252 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:37.729216 4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bnxlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 04 04:56:37.729658 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:37.729277 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" podUID="a91d185a-8a3f-4a08-9395-72e9a92265c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 04 04:56:38.728293 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:38.728261 4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bnxlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 04 04:56:38.728497 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:38.728323 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" podUID="a91d185a-8a3f-4a08-9395-72e9a92265c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 04 04:56:39.729166 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:39.729130 4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bnxlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 04 04:56:39.729565 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:39.729191 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" podUID="a91d185a-8a3f-4a08-9395-72e9a92265c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 04 04:56:40.729139 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:40.729103 4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bnxlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 04 04:56:40.729328 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:40.729164 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" podUID="a91d185a-8a3f-4a08-9395-72e9a92265c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 04 04:56:41.729351 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:41.729320 4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bnxlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 04 04:56:41.731688 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:41.729386 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" podUID="a91d185a-8a3f-4a08-9395-72e9a92265c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 04 04:56:43.590166 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:43.590099 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:43.590649 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:43.590621 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:43.591567 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:43.591545 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 04:56:44.148855 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:44.148826 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:56:44.172429 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:44.172400 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-74f5787476-n6s84"] Nov 04 04:56:44.176174 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:44.176151 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-74f5787476-n6s84" Nov 04 04:56:44.229013 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:44.228963 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 04 04:56:44.229205 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:44.229041 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 04 04:56:44.229205 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:44.229188 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-g8rvz\"" Nov 04 04:56:44.234706 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:44.234678 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-74f5787476-n6s84"] Nov 04 04:56:44.269055 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:44.269025 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw555\" (UniqueName: \"kubernetes.io/projected/0c966b96-7ebc-4e2b-98a3-8e7955f142fd-kube-api-access-cw555\") pod \"ai-gateway-controller-74f5787476-n6s84\" (UID: \"0c966b96-7ebc-4e2b-98a3-8e7955f142fd\") " pod="opendatahub/ai-gateway-controller-74f5787476-n6s84" Nov 04 04:56:44.269192 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:44.269074 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c966b96-7ebc-4e2b-98a3-8e7955f142fd-metrics-certs\") pod \"ai-gateway-controller-74f5787476-n6s84\" (UID: \"0c966b96-7ebc-4e2b-98a3-8e7955f142fd\") " pod="opendatahub/ai-gateway-controller-74f5787476-n6s84" Nov 04 04:56:44.354303 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:44.354274 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:56:44.370477 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:44.370447 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c966b96-7ebc-4e2b-98a3-8e7955f142fd-metrics-certs\") pod \"ai-gateway-controller-74f5787476-n6s84\" (UID: \"0c966b96-7ebc-4e2b-98a3-8e7955f142fd\") " pod="opendatahub/ai-gateway-controller-74f5787476-n6s84" Nov 04 04:56:44.370612 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:44.370598 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw555\" (UniqueName: \"kubernetes.io/projected/0c966b96-7ebc-4e2b-98a3-8e7955f142fd-kube-api-access-cw555\") pod \"ai-gateway-controller-74f5787476-n6s84\" (UID: \"0c966b96-7ebc-4e2b-98a3-8e7955f142fd\") " pod="opendatahub/ai-gateway-controller-74f5787476-n6s84" Nov 04 04:56:44.372940 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:44.372922 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c966b96-7ebc-4e2b-98a3-8e7955f142fd-metrics-certs\") pod \"ai-gateway-controller-74f5787476-n6s84\" (UID: \"0c966b96-7ebc-4e2b-98a3-8e7955f142fd\") " pod="opendatahub/ai-gateway-controller-74f5787476-n6s84" Nov 04 04:56:44.428558 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:44.428498 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw555\" (UniqueName: \"kubernetes.io/projected/0c966b96-7ebc-4e2b-98a3-8e7955f142fd-kube-api-access-cw555\") pod \"ai-gateway-controller-74f5787476-n6s84\" (UID: \"0c966b96-7ebc-4e2b-98a3-8e7955f142fd\") " pod="opendatahub/ai-gateway-controller-74f5787476-n6s84" Nov 04 04:56:44.488681 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:44.488651 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-74f5787476-n6s84" Nov 04 04:56:44.859773 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:44.859750 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-74f5787476-n6s84"] Nov 04 04:56:44.862695 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:56:44.862668 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c966b96_7ebc_4e2b_98a3_8e7955f142fd.slice/crio-3280289694d14f507eb8f487e016f12d37c622b5b8af2db82fa0fd00f0d88374 WatchSource:0}: Error finding container 3280289694d14f507eb8f487e016f12d37c622b5b8af2db82fa0fd00f0d88374: Status 404 returned error can't find the container with id 3280289694d14f507eb8f487e016f12d37c622b5b8af2db82fa0fd00f0d88374 Nov 04 04:56:44.874907 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:44.874879 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-74f5787476-n6s84" event={"ID":"0c966b96-7ebc-4e2b-98a3-8e7955f142fd","Type":"ContainerStarted","Data":"3280289694d14f507eb8f487e016f12d37c622b5b8af2db82fa0fd00f0d88374"} Nov 04 04:56:45.385975 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.385945 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6"] Nov 04 04:56:45.390571 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.390552 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.481001 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.480964 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.481165 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.481020 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.481165 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.481082 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.481250 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.481179 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.481250 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.481233 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49r9w\" (UniqueName: \"kubernetes.io/projected/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-kube-api-access-49r9w\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.481330 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.481251 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.481366 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.481335 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.481366 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.481358 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.481437 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.481404 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.497672 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.497639 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6"] Nov 04 04:56:45.582700 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.582670 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49r9w\" (UniqueName: \"kubernetes.io/projected/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-kube-api-access-49r9w\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.582860 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.582707 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.582860 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.582833 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.582945 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.582878 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.582945 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.582936 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.583088 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.583018 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.583163 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.583137 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.583221 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.583182 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.583276 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.583235 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.583360 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.583280 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.583360 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.583348 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.583433 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.583359 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.583467 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.583437 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.583877 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.583861 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.585163 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.585146 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.585588 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.585571 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.641277 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.641220 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.641431 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.641414 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49r9w\" (UniqueName: \"kubernetes.io/projected/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-kube-api-access-49r9w\") pod \"maas-default-gateway-openshift-default-6886b96f9-9tvb6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:45.703686 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:45.703662 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:46.130107 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:46.130078 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6"] Nov 04 04:56:46.131592 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:56:46.131565 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510d29ba_d057_4f6b_af44_aa85ab0c4ed6.slice/crio-3cfd8451a206acb80e7a0c59028d4626a37e2242c898fb0cf3f14d4cc0d1755e WatchSource:0}: Error finding container 3cfd8451a206acb80e7a0c59028d4626a37e2242c898fb0cf3f14d4cc0d1755e: Status 404 returned error can't find the container with id 3cfd8451a206acb80e7a0c59028d4626a37e2242c898fb0cf3f14d4cc0d1755e Nov 04 04:56:46.134694 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:46.134663 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:56:46.134763 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:46.134725 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:56:46.134763 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:46.134752 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:56:46.892274 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:46.892244 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" event={"ID":"510d29ba-d057-4f6b-af44-aa85ab0c4ed6","Type":"ContainerStarted","Data":"9ba1f9573a3f0a49bdb5ae38cdd71b729b4ab628b844fe9e3729eaa60e04b0a0"} Nov 04 04:56:46.892663 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:46.892636 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" event={"ID":"510d29ba-d057-4f6b-af44-aa85ab0c4ed6","Type":"ContainerStarted","Data":"3cfd8451a206acb80e7a0c59028d4626a37e2242c898fb0cf3f14d4cc0d1755e"} Nov 04 04:56:47.704001 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:47.703955 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:47.705737 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:47.705707 4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9tvb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 04 04:56:47.705882 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:47.705766 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" podUID="510d29ba-d057-4f6b-af44-aa85ab0c4ed6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 04 04:56:47.903279 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:56:47.903246 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 04:56:48.704153 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:48.704114 4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9tvb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 04 04:56:48.704651 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:48.704170 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" podUID="510d29ba-d057-4f6b-af44-aa85ab0c4ed6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 04 04:56:49.704558 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:49.704525 4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9tvb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 04 04:56:49.704960 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:49.704578 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" podUID="510d29ba-d057-4f6b-af44-aa85ab0c4ed6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 04 04:56:50.704891 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:50.704849 4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9tvb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 04 04:56:50.705377 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:50.704903 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" podUID="510d29ba-d057-4f6b-af44-aa85ab0c4ed6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 04 04:56:51.704865 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:51.704829 4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9tvb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 04 04:56:51.705073 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:51.704880 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" podUID="510d29ba-d057-4f6b-af44-aa85ab0c4ed6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 04 04:56:51.918823 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:51.918787 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-74f5787476-n6s84" event={"ID":"0c966b96-7ebc-4e2b-98a3-8e7955f142fd","Type":"ContainerStarted","Data":"085ff65741083a379823dbe858f21b09a10ebc8df4a728b9c61e90154f0db515"} Nov 04 04:56:51.919044 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:51.918876 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-74f5787476-n6s84" Nov 04 04:56:52.039799 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:52.039684 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-74f5787476-n6s84" podStartSLOduration=1.6014924019999999 podStartE2EDuration="8.039664723s" totalImagesPullingTime="6.438172321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:56:52.038545567 +0000 UTC m=+570.929402586" watchObservedRunningTime="2026-11-04 04:56:52.039664723 +0000 UTC m=+570.930521746" Nov 04 04:56:52.040453 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:52.040425 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" podStartSLOduration=7.040416125 podStartE2EDuration="7.040416125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:56:47.08136094 +0000 UTC m=+565.972217951" watchObservedRunningTime="2026-11-04 04:56:52.040416125 +0000 UTC m=+570.931273144" Nov 04 04:56:52.704757 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:52.704717 4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9tvb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 04 04:56:52.704937 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:52.704787 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" podUID="510d29ba-d057-4f6b-af44-aa85ab0c4ed6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 04 04:56:53.704236 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:53.704200 4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9tvb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 04 04:56:53.704720 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:53.704256 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" podUID="510d29ba-d057-4f6b-af44-aa85ab0c4ed6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 04 04:56:54.754205 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:54.754175 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:54.754659 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:54.754617 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:54.755404 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:54.755386 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 04:56:55.271437 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:55.271409 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:56:55.435916 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:55.435882 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:56:55.862358 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:55.862321 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-xvj6h"] Nov 04 04:56:55.865875 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:55.865858 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-xvj6h" Nov 04 04:56:55.919223 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:55.919198 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 04 04:56:55.919398 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:55.919290 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-ctwfx\"" Nov 04 04:56:55.926033 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:55.925944 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-xvj6h"] Nov 04 04:56:55.995324 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:55.995292 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mnnd\" (UniqueName: \"kubernetes.io/projected/a97545a8-870a-4ce9-907f-069b73552821-kube-api-access-7mnnd\") pod \"payload-processing-7bb9489676-xvj6h\" (UID: \"a97545a8-870a-4ce9-907f-069b73552821\") " pod="openshift-ingress/payload-processing-7bb9489676-xvj6h" Nov 04 04:56:55.995519 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:55.995496 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a97545a8-870a-4ce9-907f-069b73552821-plugins-config-volume\") pod \"payload-processing-7bb9489676-xvj6h\" (UID: \"a97545a8-870a-4ce9-907f-069b73552821\") " pod="openshift-ingress/payload-processing-7bb9489676-xvj6h" Nov 04 04:56:56.096650 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:56.096594 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mnnd\" (UniqueName: \"kubernetes.io/projected/a97545a8-870a-4ce9-907f-069b73552821-kube-api-access-7mnnd\") pod \"payload-processing-7bb9489676-xvj6h\" (UID: \"a97545a8-870a-4ce9-907f-069b73552821\") " pod="openshift-ingress/payload-processing-7bb9489676-xvj6h" Nov 04 04:56:56.096844 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:56.096680 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a97545a8-870a-4ce9-907f-069b73552821-plugins-config-volume\") pod \"payload-processing-7bb9489676-xvj6h\" (UID: \"a97545a8-870a-4ce9-907f-069b73552821\") " pod="openshift-ingress/payload-processing-7bb9489676-xvj6h" Nov 04 04:56:56.097401 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:56.097382 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a97545a8-870a-4ce9-907f-069b73552821-plugins-config-volume\") pod \"payload-processing-7bb9489676-xvj6h\" (UID: \"a97545a8-870a-4ce9-907f-069b73552821\") " pod="openshift-ingress/payload-processing-7bb9489676-xvj6h" Nov 04 04:56:56.154872 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:56.154801 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mnnd\" (UniqueName: \"kubernetes.io/projected/a97545a8-870a-4ce9-907f-069b73552821-kube-api-access-7mnnd\") pod \"payload-processing-7bb9489676-xvj6h\" (UID: \"a97545a8-870a-4ce9-907f-069b73552821\") " pod="openshift-ingress/payload-processing-7bb9489676-xvj6h" Nov 04 04:56:56.181818 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:56.181793 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-xvj6h" Nov 04 04:56:56.559093 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:56.559069 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-xvj6h"] Nov 04 04:56:56.561422 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:56:56.561381 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97545a8_870a_4ce9_907f_069b73552821.slice/crio-9238f3cfd83291278c722b5a02ac7c5bca70c6b70bd5872ae28f5106c378b896 WatchSource:0}: Error finding container 9238f3cfd83291278c722b5a02ac7c5bca70c6b70bd5872ae28f5106c378b896: Status 404 returned error can't find the container with id 9238f3cfd83291278c722b5a02ac7c5bca70c6b70bd5872ae28f5106c378b896 Nov 04 04:56:56.947475 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:56.947436 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-xvj6h" event={"ID":"a97545a8-870a-4ce9-907f-069b73552821","Type":"ContainerStarted","Data":"9238f3cfd83291278c722b5a02ac7c5bca70c6b70bd5872ae28f5106c378b896"} Nov 04 04:56:59.389605 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:59.389486 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:56:59.965186 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:59.965150 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-xvj6h" event={"ID":"a97545a8-870a-4ce9-907f-069b73552821","Type":"ContainerStarted","Data":"6a484196ece53cf96bddf320610b5c2ddae925cfdf5bbf2b70d2fabe61fd5902"} Nov 04 04:56:59.965391 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:56:59.965210 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-xvj6h" Nov 04 04:57:00.080057 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:00.080003 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-xvj6h" podStartSLOduration=2.132687556 podStartE2EDuration="5.079971391s" totalImagesPullingTime="2.947283835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:57:00.079074535 +0000 UTC m=+578.969931555" watchObservedRunningTime="2026-11-04 04:57:00.079971391 +0000 UTC m=+578.970828411" Nov 04 04:57:02.927275 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:02.927205 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-74f5787476-n6s84" Nov 04 04:57:10.973418 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:10.973386 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-xvj6h" Nov 04 04:57:17.886281 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:57:17.886241 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 04:57:18.733428 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:18.733398 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-74f5787476-n6s84"] Nov 04 04:57:18.733659 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:18.733633 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-74f5787476-n6s84" podUID="0c966b96-7ebc-4e2b-98a3-8e7955f142fd" containerName="manager" containerID="cri-o://085ff65741083a379823dbe858f21b09a10ebc8df4a728b9c61e90154f0db515" gracePeriod=30 Nov 04 04:57:18.990111 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:18.990045 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-74f5787476-n6s84" Nov 04 04:57:19.029886 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:19.029858 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c966b96-7ebc-4e2b-98a3-8e7955f142fd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c966b96-7ebc-4e2b-98a3-8e7955f142fd-metrics-certs\") pod \"0c966b96-7ebc-4e2b-98a3-8e7955f142fd\" (UID: \"0c966b96-7ebc-4e2b-98a3-8e7955f142fd\") " Nov 04 04:57:19.030060 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:19.029918 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c966b96-7ebc-4e2b-98a3-8e7955f142fd-kube-api-access-cw555\" (UniqueName: \"kubernetes.io/projected/0c966b96-7ebc-4e2b-98a3-8e7955f142fd-kube-api-access-cw555\") pod \"0c966b96-7ebc-4e2b-98a3-8e7955f142fd\" (UID: \"0c966b96-7ebc-4e2b-98a3-8e7955f142fd\") " Nov 04 04:57:19.032187 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:19.032153 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c966b96-7ebc-4e2b-98a3-8e7955f142fd-kube-api-access-cw555" pod "0c966b96-7ebc-4e2b-98a3-8e7955f142fd" (UID: "0c966b96-7ebc-4e2b-98a3-8e7955f142fd"). InnerVolumeSpecName "kube-api-access-cw555". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:57:19.032316 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:19.032238 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c966b96-7ebc-4e2b-98a3-8e7955f142fd-metrics-certs" pod "0c966b96-7ebc-4e2b-98a3-8e7955f142fd" (UID: "0c966b96-7ebc-4e2b-98a3-8e7955f142fd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:57:19.060239 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:19.060208 4581 generic.go:356] "Generic (PLEG): container finished" podID="0c966b96-7ebc-4e2b-98a3-8e7955f142fd" containerID="085ff65741083a379823dbe858f21b09a10ebc8df4a728b9c61e90154f0db515" exitCode=0 Nov 04 04:57:19.060395 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:19.060256 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-74f5787476-n6s84" Nov 04 04:57:19.060395 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:19.060279 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-74f5787476-n6s84" event={"ID":"0c966b96-7ebc-4e2b-98a3-8e7955f142fd","Type":"ContainerDied","Data":"085ff65741083a379823dbe858f21b09a10ebc8df4a728b9c61e90154f0db515"} Nov 04 04:57:19.060395 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:19.060335 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-74f5787476-n6s84" event={"ID":"0c966b96-7ebc-4e2b-98a3-8e7955f142fd","Type":"ContainerDied","Data":"3280289694d14f507eb8f487e016f12d37c622b5b8af2db82fa0fd00f0d88374"} Nov 04 04:57:19.060395 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:19.060358 4581 scope.go:122] "RemoveContainer" containerID="085ff65741083a379823dbe858f21b09a10ebc8df4a728b9c61e90154f0db515" Nov 04 04:57:19.070237 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:19.070215 4581 scope.go:122] "RemoveContainer" containerID="085ff65741083a379823dbe858f21b09a10ebc8df4a728b9c61e90154f0db515" Nov 04 04:57:19.070481 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:57:19.070462 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"085ff65741083a379823dbe858f21b09a10ebc8df4a728b9c61e90154f0db515\": container with ID starting with 085ff65741083a379823dbe858f21b09a10ebc8df4a728b9c61e90154f0db515 not found: ID does not exist" containerID="085ff65741083a379823dbe858f21b09a10ebc8df4a728b9c61e90154f0db515" Nov 04 04:57:19.070534 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:19.070492 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"085ff65741083a379823dbe858f21b09a10ebc8df4a728b9c61e90154f0db515"} err="failed to get container status \"085ff65741083a379823dbe858f21b09a10ebc8df4a728b9c61e90154f0db515\": rpc error: code = NotFound desc = could not find container \"085ff65741083a379823dbe858f21b09a10ebc8df4a728b9c61e90154f0db515\": container with ID starting with 085ff65741083a379823dbe858f21b09a10ebc8df4a728b9c61e90154f0db515 not found: ID does not exist" Nov 04 04:57:19.130557 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:19.130526 4581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c966b96-7ebc-4e2b-98a3-8e7955f142fd-metrics-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:57:19.130557 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:19.130552 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw555\" (UniqueName: \"kubernetes.io/projected/0c966b96-7ebc-4e2b-98a3-8e7955f142fd-kube-api-access-cw555\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:57:19.217969 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:19.217926 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-74f5787476-n6s84"] Nov 04 04:57:19.229495 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:19.229469 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-74f5787476-n6s84"] Nov 04 04:57:21.231164 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:21.231134 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c966b96-7ebc-4e2b-98a3-8e7955f142fd" path="/var/lib/kubelet/pods/0c966b96-7ebc-4e2b-98a3-8e7955f142fd/volumes" Nov 04 04:57:30.433476 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.433443 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7"] Nov 04 04:57:30.433892 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.433879 4581 cpu_manager.go:414] "containerMap: removing container" podUID="0c966b96-7ebc-4e2b-98a3-8e7955f142fd" containerName="manager" Nov 04 04:57:30.433940 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.433894 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c966b96-7ebc-4e2b-98a3-8e7955f142fd" containerName="manager" Nov 04 04:57:30.434011 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.433979 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c966b96-7ebc-4e2b-98a3-8e7955f142fd" containerName="manager" Nov 04 04:57:30.437697 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.437683 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.539067 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.539029 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.539261 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.539074 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.539261 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.539104 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.539261 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.539196 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.539261 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.539217 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.539261 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.539255 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.539553 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.539349 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.539553 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.539378 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.539553 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.539400 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgggv\" (UniqueName: \"kubernetes.io/projected/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-kube-api-access-cgggv\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.549647 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.549619 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:57:30.640490 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.640461 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.640490 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.640500 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.640745 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.640524 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.640745 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.640560 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.640745 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.640587 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.640745 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.640623 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.640745 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.640656 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.640745 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.640674 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.640745 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.640691 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgggv\" (UniqueName: \"kubernetes.io/projected/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-kube-api-access-cgggv\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.641377 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.641351 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.641625 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.641576 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.641786 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.641751 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.642105 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.642080 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.642203 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.642083 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.642491 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.642456 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7"] Nov 04 04:57:30.644593 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.644566 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.645263 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.645236 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.698436 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.698358 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgggv\" (UniqueName: \"kubernetes.io/projected/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-kube-api-access-cgggv\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.698436 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.698370 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-vnmc7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:30.750315 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:30.750290 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:31.018365 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:31.018288 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:57:31.177573 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:31.177543 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7"] Nov 04 04:57:31.219770 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:57:31.219739 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2079faa0_1dfe_43f2_8f3a_199e92ff79e7.slice/crio-d2f1988c837ef509ed96323801fecc9ce994ea8d58de11287d5f8008a2ed57a7 WatchSource:0}: Error finding container d2f1988c837ef509ed96323801fecc9ce994ea8d58de11287d5f8008a2ed57a7: Status 404 returned error can't find the container with id d2f1988c837ef509ed96323801fecc9ce994ea8d58de11287d5f8008a2ed57a7 Nov 04 04:57:31.222001 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:31.221945 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:57:31.222123 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:31.222047 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:57:31.222188 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:31.222120 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:57:32.126959 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:32.126924 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" event={"ID":"2079faa0-1dfe-43f2-8f3a-199e92ff79e7","Type":"ContainerStarted","Data":"8ce9aa27131079b1f4048ccbf4478c21c5fa6084d56f025ffa6c3291646d2813"} Nov 04 04:57:32.127390 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:32.126981 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" event={"ID":"2079faa0-1dfe-43f2-8f3a-199e92ff79e7","Type":"ContainerStarted","Data":"d2f1988c837ef509ed96323801fecc9ce994ea8d58de11287d5f8008a2ed57a7"} Nov 04 04:57:32.297825 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:32.297766 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" podStartSLOduration=2.297750061 podStartE2EDuration="2.297750061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:57:32.295625576 +0000 UTC m=+611.186482596" watchObservedRunningTime="2026-11-04 04:57:32.297750061 +0000 UTC m=+611.188607081" Nov 04 04:57:32.705576 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:32.705541 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:57:32.750949 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:32.750919 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:32.775155 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:32.775119 4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vnmc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 04 04:57:32.775439 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:32.775407 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" podUID="2079faa0-1dfe-43f2-8f3a-199e92ff79e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 04 04:57:32.842520 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:32.842477 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:57:33.751467 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:33.751434 4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vnmc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 04 04:57:33.751953 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:33.751484 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" podUID="2079faa0-1dfe-43f2-8f3a-199e92ff79e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 04 04:57:34.387670 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:34.387632 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:57:34.521457 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:34.521418 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:57:34.751292 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:34.751211 4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vnmc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 04 04:57:34.751292 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:34.751265 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" podUID="2079faa0-1dfe-43f2-8f3a-199e92ff79e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 04 04:57:35.751482 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:35.751451 4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vnmc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 04 04:57:35.751884 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:35.751507 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" podUID="2079faa0-1dfe-43f2-8f3a-199e92ff79e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 04 04:57:36.751290 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:36.751255 4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vnmc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 04 04:57:36.751464 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:36.751315 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" podUID="2079faa0-1dfe-43f2-8f3a-199e92ff79e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 04 04:57:37.750883 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:37.750852 4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vnmc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 04 04:57:37.751291 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:37.750901 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" podUID="2079faa0-1dfe-43f2-8f3a-199e92ff79e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 04 04:57:38.751339 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:38.751294 4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vnmc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 04 04:57:38.751819 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:38.751357 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" podUID="2079faa0-1dfe-43f2-8f3a-199e92ff79e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 04 04:57:39.757655 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:39.757627 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:39.758159 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:39.758062 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:39.758968 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:39.758945 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 04:57:47.885386 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:57:47.885350 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 04:57:59.735199 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:57:59.735162 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:17.239329 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:17.239302 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:17.358692 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:17.358638 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:17.775956 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:17.775914 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n"] Nov 04 04:58:17.779618 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:17.779594 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:17.847353 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:17.847309 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:17.879880 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:17.879841 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n"] Nov 04 04:58:17.886567 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:17.885252 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-370078-openshift-default-dockercfg-r9m9c\"" Nov 04 04:58:17.888599 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:58:17.888566 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 04:58:17.926511 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:17.926480 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-token\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:17.926776 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:17.926522 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-data\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:17.926776 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:17.926550 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istiod-ca-cert\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:17.926776 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:17.926657 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-credential-socket\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:17.926776 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:17.926713 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-podinfo\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:17.926776 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:17.926739 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlw65\" (UniqueName: \"kubernetes.io/projected/45926cc8-a8ed-40b8-a5f0-db1336dcc102-kube-api-access-dlw65\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:17.927080 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:17.926791 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-workload-certs\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:17.927080 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:17.926853 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-envoy\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:17.927080 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:17.926894 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-workload-socket\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:18.027737 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.027660 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-envoy\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:18.027737 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.027695 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-workload-socket\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:18.027960 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.027749 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-token\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:18.027960 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.027771 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-data\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:18.027960 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.027791 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istiod-ca-cert\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:18.027960 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.027808 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-credential-socket\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:18.027960 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.027825 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-podinfo\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:18.027960 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.027840 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlw65\" (UniqueName: \"kubernetes.io/projected/45926cc8-a8ed-40b8-a5f0-db1336dcc102-kube-api-access-dlw65\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:18.027960 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.027921 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-workload-certs\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:18.028395 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.028310 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-credential-socket\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:18.028483 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.028451 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-workload-certs\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:18.028483 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.028469 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-data\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:18.028581 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.028566 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-workload-socket\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:18.028869 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.028844 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istiod-ca-cert\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:18.030424 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.030397 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-envoy\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:18.030698 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.030680 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-podinfo\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:18.054440 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.054413 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:18.090329 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.090297 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-token\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:18.090469 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.090452 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlw65\" (UniqueName: \"kubernetes.io/projected/45926cc8-a8ed-40b8-a5f0-db1336dcc102-kube-api-access-dlw65\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-hp82n\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:18.093312 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.093294 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:18.179061 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.178783 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:18.380404 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.380369 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm"] Nov 04 04:58:18.384276 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.384256 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.436706 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.436679 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-1874d8-openshift-default-dockercfg-tcccq\"" Nov 04 04:58:18.442960 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.442931 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm"] Nov 04 04:58:18.466288 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.466254 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:18.534857 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.534823 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-podinfo\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.535074 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.534866 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istiod-ca-cert\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.535074 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.534897 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b8cq\" (UniqueName: \"kubernetes.io/projected/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-kube-api-access-4b8cq\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.535074 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.534935 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-workload-certs\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.535074 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.534953 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-data\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.535074 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.535038 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-envoy\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.535074 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.535062 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-credential-socket\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.535388 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.535109 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-token\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.535388 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.535190 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-workload-socket\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.558717 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.558689 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n"] Nov 04 04:58:18.560299 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:58:18.560272 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45926cc8_a8ed_40b8_a5f0_db1336dcc102.slice/crio-72a1fbc84cf392a42b1497d6c5703454bb506e37242ccf5f29c258ce5e82b575 WatchSource:0}: Error finding container 72a1fbc84cf392a42b1497d6c5703454bb506e37242ccf5f29c258ce5e82b575: Status 404 returned error can't find the container with id 72a1fbc84cf392a42b1497d6c5703454bb506e37242ccf5f29c258ce5e82b575 Nov 04 04:58:18.562607 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.562563 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:58:18.562670 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.562655 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:58:18.562720 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.562708 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:58:18.635890 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.635797 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-credential-socket\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.635890 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.635857 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-token\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.635890 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.635888 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-workload-socket\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.636181 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.635934 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-podinfo\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.636181 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.635969 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istiod-ca-cert\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.636181 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.636028 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b8cq\" (UniqueName: \"kubernetes.io/projected/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-kube-api-access-4b8cq\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.636181 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.636089 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-workload-certs\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.636181 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.636115 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-data\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.636181 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.636183 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-envoy\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.636501 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.636294 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-credential-socket\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.636588 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.636561 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-workload-certs\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.636892 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.636871 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istiod-ca-cert\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.637003 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.636970 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-workload-socket\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.637080 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.636968 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-data\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.638538 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.638511 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-envoy\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.639287 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.639265 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-podinfo\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.666005 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.665953 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:18.703942 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.703902 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-token\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.704588 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.704569 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b8cq\" (UniqueName: \"kubernetes.io/projected/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-kube-api-access-4b8cq\") pod \"e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:18.796327 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.796284 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:18.944851 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.944710 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:18.997613 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:18.997588 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:19.108825 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:19.108790 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:19.285475 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:19.285401 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:19.357806 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:19.357762 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" event={"ID":"45926cc8-a8ed-40b8-a5f0-db1336dcc102","Type":"ContainerStarted","Data":"95c9490826687caa0924e3da386f02b17adf3f868cfa41e36fe1a01f0c106350"} Nov 04 04:58:19.357979 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:19.357831 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" event={"ID":"45926cc8-a8ed-40b8-a5f0-db1336dcc102","Type":"ContainerStarted","Data":"72a1fbc84cf392a42b1497d6c5703454bb506e37242ccf5f29c258ce5e82b575"} Nov 04 04:58:19.425263 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:19.425068 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:19.426709 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:19.426685 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm"] Nov 04 04:58:19.428026 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:58:19.427980 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5ae17b_aca8_4988_8ed1_7042c62fc32f.slice/crio-aa5fb2aaf2855ecc810c13534fd0aaa97f2e9581eae91c0a0b4b4b1407bc1cf1 WatchSource:0}: Error finding container aa5fb2aaf2855ecc810c13534fd0aaa97f2e9581eae91c0a0b4b4b1407bc1cf1: Status 404 returned error can't find the container with id aa5fb2aaf2855ecc810c13534fd0aaa97f2e9581eae91c0a0b4b4b1407bc1cf1 Nov 04 04:58:19.430439 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:19.430408 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:58:19.430553 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:19.430471 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:58:19.430553 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:19.430499 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:58:19.477429 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:19.477373 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" podStartSLOduration=2.477355635 podStartE2EDuration="2.477355635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:58:19.476539931 +0000 UTC m=+658.367396978" watchObservedRunningTime="2026-11-04 04:58:19.477355635 +0000 UTC m=+658.368212656" Nov 04 04:58:19.706406 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:19.706366 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:20.094245 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:20.094168 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:20.099549 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:20.099527 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:20.364582 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:20.364550 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" event={"ID":"ad5ae17b-aca8-4988-8ed1-7042c62fc32f","Type":"ContainerStarted","Data":"8098505851b03a6fe65a0014515ea965c1f6e89ce749aa093d59401232c0d2dc"} Nov 04 04:58:20.364757 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:20.364605 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" event={"ID":"ad5ae17b-aca8-4988-8ed1-7042c62fc32f","Type":"ContainerStarted","Data":"aa5fb2aaf2855ecc810c13534fd0aaa97f2e9581eae91c0a0b4b4b1407bc1cf1"} Nov 04 04:58:20.364842 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:20.364796 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:20.365623 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:20.365605 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:58:20.484712 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:20.484661 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" podStartSLOduration=2.484642617 podStartE2EDuration="2.484642617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:58:20.483181743 +0000 UTC m=+659.374038797" watchObservedRunningTime="2026-11-04 04:58:20.484642617 +0000 UTC m=+659.375499635" Nov 04 04:58:20.998628 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:20.998597 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:21.003935 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:21.003909 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:21.370227 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:21.370193 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:21.371052 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:21.371034 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:58:22.918202 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:22.918167 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:23.113162 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:23.113130 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:23.243502 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:23.243417 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:23.545337 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:23.545261 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:23.767379 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:23.767344 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:23.891857 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:23.891825 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:24.225962 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:24.225877 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:24.389921 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:24.389885 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:24.512811 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:24.512721 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:26.399340 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:26.399313 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:26.571119 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:26.571086 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:26.825004 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:26.824899 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc"] Nov 04 04:58:26.828758 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:26.828742 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc" Nov 04 04:58:26.926079 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:26.926042 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc\" (UID: \"bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc" Nov 04 04:58:26.926288 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:26.926108 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjr6n\" (UniqueName: \"kubernetes.io/projected/bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e-kube-api-access-zjr6n\") pod \"payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc\" (UID: \"bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc" Nov 04 04:58:26.932918 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:26.932888 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-4cce33\"" Nov 04 04:58:26.933168 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:26.933001 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-4cce33-dockercfg-zgtwj\"" Nov 04 04:58:26.935404 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:26.935382 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc"] Nov 04 04:58:27.027491 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:27.027459 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc\" (UID: \"bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc" Nov 04 04:58:27.027672 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:27.027501 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjr6n\" (UniqueName: \"kubernetes.io/projected/bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e-kube-api-access-zjr6n\") pod \"payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc\" (UID: \"bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc" Nov 04 04:58:27.028322 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:27.028299 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc\" (UID: \"bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc" Nov 04 04:58:27.085858 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:27.085778 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjr6n\" (UniqueName: \"kubernetes.io/projected/bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e-kube-api-access-zjr6n\") pod \"payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc\" (UID: \"bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc" Nov 04 04:58:27.141785 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:27.141757 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc" Nov 04 04:58:27.235405 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:27.235371 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:27.521941 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:27.521919 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc"] Nov 04 04:58:27.524287 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:58:27.524256 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc2f10e9_0b97_4522_bc6f_c8ac8e4ef30e.slice/crio-f27ecf91f4730b6ccf2b0d2d32448af53280b41c1a0d3382cf1f40a1cfb0a32c WatchSource:0}: Error finding container f27ecf91f4730b6ccf2b0d2d32448af53280b41c1a0d3382cf1f40a1cfb0a32c: Status 404 returned error can't find the container with id f27ecf91f4730b6ccf2b0d2d32448af53280b41c1a0d3382cf1f40a1cfb0a32c Nov 04 04:58:27.777829 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:27.777732 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285"] Nov 04 04:58:27.781608 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:27.781584 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:27.845544 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:27.845509 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285"] Nov 04 04:58:27.858209 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:27.858180 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:27.859426 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:27.859409 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-0aae2f-openshift-default-dockercfg-9n62j\"" Nov 04 04:58:27.937047 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:27.937006 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-credential-socket\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:27.937249 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:27.937116 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-data\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:27.937249 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:27.937237 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-workload-certs\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:27.937404 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:27.937386 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-workload-socket\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:27.937502 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:27.937442 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istiod-ca-cert\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:27.937502 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:27.937473 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjxcs\" (UniqueName: \"kubernetes.io/projected/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-kube-api-access-fjxcs\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:27.937652 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:27.937551 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-envoy\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:27.937652 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:27.937583 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-podinfo\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:27.937652 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:27.937628 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-token\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:28.038796 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.038715 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-workload-certs\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:28.038955 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.038808 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-workload-socket\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:28.038955 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.038838 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istiod-ca-cert\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:28.038955 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.038863 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjxcs\" (UniqueName: \"kubernetes.io/projected/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-kube-api-access-fjxcs\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:28.039168 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.039046 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-envoy\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:28.039168 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.039082 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-podinfo\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:28.039168 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.039113 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-token\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:28.039168 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.039141 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-credential-socket\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:28.039626 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.039179 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-data\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:28.039626 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.039194 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-workload-certs\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:28.039626 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.039244 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-workload-socket\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:28.039626 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.039512 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-credential-socket\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:28.039626 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.039605 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-data\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:28.039804 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.039634 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istiod-ca-cert\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:28.041571 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.041553 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-envoy\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:28.041658 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.041641 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-podinfo\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:28.096713 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.096689 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjxcs\" (UniqueName: \"kubernetes.io/projected/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-kube-api-access-fjxcs\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:28.096856 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.096694 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-token\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:28.102876 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.102844 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:28.394823 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.394788 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:28.405844 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.405809 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc" event={"ID":"bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e","Type":"ContainerStarted","Data":"8bbb1945aeb28d145c6a8da934685d05df7afef6f125baa1cc83be31a53b7ecc"} Nov 04 04:58:28.406074 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.406004 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc" Nov 04 04:58:28.406074 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.406037 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc" event={"ID":"bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e","Type":"ContainerStarted","Data":"f27ecf91f4730b6ccf2b0d2d32448af53280b41c1a0d3382cf1f40a1cfb0a32c"} Nov 04 04:58:28.519868 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.519776 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc" podStartSLOduration=2.519757978 podStartE2EDuration="2.519757978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:58:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:58:28.519589381 +0000 UTC m=+667.410446403" watchObservedRunningTime="2026-11-04 04:58:28.519757978 +0000 UTC m=+667.410614998" Nov 04 04:58:28.819781 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.819759 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285"] Nov 04 04:58:28.821818 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:58:28.821783 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa8dcd7f_0f9f_4eef_a831_8d164c92830e.slice/crio-219fb0413d8880e02ecfff33866226caf2269cd96672debc91394a8d08b8d893 WatchSource:0}: Error finding container 219fb0413d8880e02ecfff33866226caf2269cd96672debc91394a8d08b8d893: Status 404 returned error can't find the container with id 219fb0413d8880e02ecfff33866226caf2269cd96672debc91394a8d08b8d893 Nov 04 04:58:28.824336 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.824301 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:58:28.824433 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.824365 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:58:28.824433 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:28.824394 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:58:29.039439 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:29.039400 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln"] Nov 04 04:58:29.043204 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:29.043182 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" Nov 04 04:58:29.098533 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:29.098460 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-3eb31c-serving-cert\"" Nov 04 04:58:29.101358 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:29.101330 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln"] Nov 04 04:58:29.151293 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:29.151267 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t98nf\" (UniqueName: \"kubernetes.io/projected/0e8292c4-8027-4026-ba22-efc5004ff544-kube-api-access-t98nf\") pod \"maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln\" (UID: \"0e8292c4-8027-4026-ba22-efc5004ff544\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" Nov 04 04:58:29.151451 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:29.151347 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e8292c4-8027-4026-ba22-efc5004ff544-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln\" (UID: \"0e8292c4-8027-4026-ba22-efc5004ff544\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" Nov 04 04:58:29.151451 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:29.151400 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e8292c4-8027-4026-ba22-efc5004ff544-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln\" (UID: \"0e8292c4-8027-4026-ba22-efc5004ff544\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" Nov 04 04:58:29.151532 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:29.151485 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e8292c4-8027-4026-ba22-efc5004ff544-metrics-certs\") pod \"maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln\" (UID: \"0e8292c4-8027-4026-ba22-efc5004ff544\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" Nov 04 04:58:29.225719 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:29.225684 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:29.252345 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:29.252308 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e8292c4-8027-4026-ba22-efc5004ff544-metrics-certs\") pod \"maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln\" (UID: \"0e8292c4-8027-4026-ba22-efc5004ff544\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" Nov 04 04:58:29.252515 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:29.252421 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t98nf\" (UniqueName: \"kubernetes.io/projected/0e8292c4-8027-4026-ba22-efc5004ff544-kube-api-access-t98nf\") pod \"maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln\" (UID: \"0e8292c4-8027-4026-ba22-efc5004ff544\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" Nov 04 04:58:29.252585 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:29.252519 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e8292c4-8027-4026-ba22-efc5004ff544-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln\" (UID: \"0e8292c4-8027-4026-ba22-efc5004ff544\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" Nov 04 04:58:29.252585 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:29.252547 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e8292c4-8027-4026-ba22-efc5004ff544-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln\" (UID: \"0e8292c4-8027-4026-ba22-efc5004ff544\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" Nov 04 04:58:29.252720 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:58:29.252702 4581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-serving-cert: secret "maas-api-e2e-worker-w4-3eb31c-serving-cert" not found Nov 04 04:58:29.252797 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:58:29.252787 4581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e8292c4-8027-4026-ba22-efc5004ff544-maas-api-tls podName:0e8292c4-8027-4026-ba22-efc5004ff544 nodeName:}" failed. No retries permitted until 2026-11-04 04:58:29.752765296 +0000 UTC m=+668.643622308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e8292c4-8027-4026-ba22-efc5004ff544-maas-api-tls") pod "maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" (UID: "0e8292c4-8027-4026-ba22-efc5004ff544") : secret "maas-api-e2e-worker-w4-3eb31c-serving-cert" not found Nov 04 04:58:29.253226 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:29.253207 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e8292c4-8027-4026-ba22-efc5004ff544-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln\" (UID: \"0e8292c4-8027-4026-ba22-efc5004ff544\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" Nov 04 04:58:29.254908 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:29.254889 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e8292c4-8027-4026-ba22-efc5004ff544-metrics-certs\") pod \"maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln\" (UID: \"0e8292c4-8027-4026-ba22-efc5004ff544\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" Nov 04 04:58:29.310039 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:29.310013 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t98nf\" (UniqueName: \"kubernetes.io/projected/0e8292c4-8027-4026-ba22-efc5004ff544-kube-api-access-t98nf\") pod \"maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln\" (UID: \"0e8292c4-8027-4026-ba22-efc5004ff544\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" Nov 04 04:58:29.412260 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:29.412223 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" event={"ID":"aa8dcd7f-0f9f-4eef-a831-8d164c92830e","Type":"ContainerStarted","Data":"33c3a7763a73a9cc67e554288a78ac02b3ae5e024361baf5d16f26a0894cc30c"} Nov 04 04:58:29.412437 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:29.412278 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" event={"ID":"aa8dcd7f-0f9f-4eef-a831-8d164c92830e","Type":"ContainerStarted","Data":"219fb0413d8880e02ecfff33866226caf2269cd96672debc91394a8d08b8d893"} Nov 04 04:58:29.505499 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:29.505466 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:29.535214 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:29.535154 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" podStartSLOduration=2.535139434 podStartE2EDuration="2.535139434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:58:29.533199601 +0000 UTC m=+668.424056622" watchObservedRunningTime="2026-11-04 04:58:29.535139434 +0000 UTC m=+668.425996454" Nov 04 04:58:29.705616 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:29.705534 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:29.759413 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:29.759382 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e8292c4-8027-4026-ba22-efc5004ff544-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln\" (UID: \"0e8292c4-8027-4026-ba22-efc5004ff544\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" Nov 04 04:58:29.761892 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:29.761868 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e8292c4-8027-4026-ba22-efc5004ff544-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln\" (UID: \"0e8292c4-8027-4026-ba22-efc5004ff544\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" Nov 04 04:58:29.955808 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:29.955728 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" Nov 04 04:58:30.329057 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:30.329028 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln"] Nov 04 04:58:30.330445 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:58:30.330413 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8292c4_8027_4026_ba22_efc5004ff544.slice/crio-9ec6a89c6a972a206f6d18cd1e254cfbf01462df7596c306f73dc461257a9d54 WatchSource:0}: Error finding container 9ec6a89c6a972a206f6d18cd1e254cfbf01462df7596c306f73dc461257a9d54: Status 404 returned error can't find the container with id 9ec6a89c6a972a206f6d18cd1e254cfbf01462df7596c306f73dc461257a9d54 Nov 04 04:58:30.331820 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:30.331803 4581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 04:58:30.395356 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:30.395317 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:30.400542 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:30.400521 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:30.418881 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:30.418846 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" event={"ID":"0e8292c4-8027-4026-ba22-efc5004ff544","Type":"ContainerStarted","Data":"9ec6a89c6a972a206f6d18cd1e254cfbf01462df7596c306f73dc461257a9d54"} Nov 04 04:58:30.419163 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:30.419147 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:30.420026 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:30.419979 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:58:31.148881 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:31.148845 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:31.324656 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:31.324624 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:31.425849 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:31.425750 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" event={"ID":"0e8292c4-8027-4026-ba22-efc5004ff544","Type":"ContainerStarted","Data":"de5678a5ee6d14e44d88ded2f59234c5fa0e569b134bdd130aa31d5df298408c"} Nov 04 04:58:31.426024 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:31.425909 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" Nov 04 04:58:31.541332 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:31.541275 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" podStartSLOduration=2.557428749 podStartE2EDuration="3.541261129s" totalImagesPullingTime="983.83238ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:58:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:58:31.540448307 +0000 UTC m=+670.431305328" watchObservedRunningTime="2026-11-04 04:58:31.541261129 +0000 UTC m=+670.432118149" Nov 04 04:58:32.093594 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:32.093508 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:32.320388 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:32.320358 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:32.445695 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:32.445667 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:33.634539 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:33.634478 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:33.802146 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:33.802110 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:33.928355 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:33.928237 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:34.087415 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:34.087352 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:34.458902 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:34.458874 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v"] Nov 04 04:58:34.462927 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:34.462906 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" Nov 04 04:58:34.516590 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:34.516561 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00ab6653-dd2e-4dd5-bf65-27428a66e07f-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v\" (UID: \"00ab6653-dd2e-4dd5-bf65-27428a66e07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" Nov 04 04:58:34.516730 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:34.516596 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00ab6653-dd2e-4dd5-bf65-27428a66e07f-maas-api-tls\") pod \"maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v\" (UID: \"00ab6653-dd2e-4dd5-bf65-27428a66e07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" Nov 04 04:58:34.516730 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:34.516618 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00ab6653-dd2e-4dd5-bf65-27428a66e07f-metrics-certs\") pod \"maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v\" (UID: \"00ab6653-dd2e-4dd5-bf65-27428a66e07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" Nov 04 04:58:34.516799 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:34.516731 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-944mw\" (UniqueName: \"kubernetes.io/projected/00ab6653-dd2e-4dd5-bf65-27428a66e07f-kube-api-access-944mw\") pod \"maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v\" (UID: \"00ab6653-dd2e-4dd5-bf65-27428a66e07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" Nov 04 04:58:34.568505 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:34.568478 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-cedfca-serving-cert\"" Nov 04 04:58:34.571278 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:34.571251 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v"] Nov 04 04:58:34.617725 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:34.617691 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00ab6653-dd2e-4dd5-bf65-27428a66e07f-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v\" (UID: \"00ab6653-dd2e-4dd5-bf65-27428a66e07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" Nov 04 04:58:34.617903 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:34.617735 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00ab6653-dd2e-4dd5-bf65-27428a66e07f-maas-api-tls\") pod \"maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v\" (UID: \"00ab6653-dd2e-4dd5-bf65-27428a66e07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" Nov 04 04:58:34.617903 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:34.617764 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00ab6653-dd2e-4dd5-bf65-27428a66e07f-metrics-certs\") pod \"maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v\" (UID: \"00ab6653-dd2e-4dd5-bf65-27428a66e07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" Nov 04 04:58:34.617903 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:34.617792 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-944mw\" (UniqueName: \"kubernetes.io/projected/00ab6653-dd2e-4dd5-bf65-27428a66e07f-kube-api-access-944mw\") pod \"maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v\" (UID: \"00ab6653-dd2e-4dd5-bf65-27428a66e07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" Nov 04 04:58:34.617903 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:58:34.617880 4581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-serving-cert: secret "maas-api-e2e-worker-w6-cedfca-serving-cert" not found Nov 04 04:58:34.618124 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:58:34.617981 4581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00ab6653-dd2e-4dd5-bf65-27428a66e07f-maas-api-tls podName:00ab6653-dd2e-4dd5-bf65-27428a66e07f nodeName:}" failed. No retries permitted until 2026-11-04 04:58:35.117958105 +0000 UTC m=+674.008815104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/00ab6653-dd2e-4dd5-bf65-27428a66e07f-maas-api-tls") pod "maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" (UID: "00ab6653-dd2e-4dd5-bf65-27428a66e07f") : secret "maas-api-e2e-worker-w6-cedfca-serving-cert" not found Nov 04 04:58:34.618429 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:34.618409 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00ab6653-dd2e-4dd5-bf65-27428a66e07f-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v\" (UID: \"00ab6653-dd2e-4dd5-bf65-27428a66e07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" Nov 04 04:58:34.620640 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:34.620614 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00ab6653-dd2e-4dd5-bf65-27428a66e07f-metrics-certs\") pod \"maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v\" (UID: \"00ab6653-dd2e-4dd5-bf65-27428a66e07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" Nov 04 04:58:34.645813 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:34.645781 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:34.675600 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:34.675574 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-944mw\" (UniqueName: \"kubernetes.io/projected/00ab6653-dd2e-4dd5-bf65-27428a66e07f-kube-api-access-944mw\") pod \"maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v\" (UID: \"00ab6653-dd2e-4dd5-bf65-27428a66e07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" Nov 04 04:58:34.809658 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:34.809580 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:34.929361 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:34.929328 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:35.123265 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:35.123235 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00ab6653-dd2e-4dd5-bf65-27428a66e07f-maas-api-tls\") pod \"maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v\" (UID: \"00ab6653-dd2e-4dd5-bf65-27428a66e07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" Nov 04 04:58:35.125746 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:35.125717 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00ab6653-dd2e-4dd5-bf65-27428a66e07f-maas-api-tls\") pod \"maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v\" (UID: \"00ab6653-dd2e-4dd5-bf65-27428a66e07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" Nov 04 04:58:35.376223 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:35.376145 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" Nov 04 04:58:35.748222 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:35.748196 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v"] Nov 04 04:58:35.749882 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:58:35.749854 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ab6653_dd2e_4dd5_bf65_27428a66e07f.slice/crio-ce017997e493c7e05b40fc032fe20b385ad182ff2ba9b644dc6fbda55f3391bc WatchSource:0}: Error finding container ce017997e493c7e05b40fc032fe20b385ad182ff2ba9b644dc6fbda55f3391bc: Status 404 returned error can't find the container with id ce017997e493c7e05b40fc032fe20b385ad182ff2ba9b644dc6fbda55f3391bc Nov 04 04:58:36.255617 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:36.255584 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:36.432580 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:36.432544 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:36.457178 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:36.457147 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" event={"ID":"00ab6653-dd2e-4dd5-bf65-27428a66e07f","Type":"ContainerStarted","Data":"ce017997e493c7e05b40fc032fe20b385ad182ff2ba9b644dc6fbda55f3391bc"} Nov 04 04:58:36.567581 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:36.567557 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:36.854709 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:36.854675 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:37.033843 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:37.033814 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:37.161595 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:37.161502 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:37.439511 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:37.439444 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" Nov 04 04:58:37.464252 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:37.464220 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" event={"ID":"00ab6653-dd2e-4dd5-bf65-27428a66e07f","Type":"ContainerStarted","Data":"91bbdad1580664b2a96d124b863ad28e1720be1e8a39024517af8bdcf5742f5e"} Nov 04 04:58:37.464383 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:37.464269 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" Nov 04 04:58:37.669534 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:37.669485 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" podStartSLOduration=2.85726498 podStartE2EDuration="3.66947137s" totalImagesPullingTime="812.20639ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:58:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:58:37.668920036 +0000 UTC m=+676.559777057" watchObservedRunningTime="2026-11-04 04:58:37.66947137 +0000 UTC m=+676.560328390" Nov 04 04:58:38.502057 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:38.502018 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:38.680099 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:38.680065 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:39.415307 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:39.415285 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc" Nov 04 04:58:39.731280 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:39.731200 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj"] Nov 04 04:58:39.734865 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:39.734845 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" Nov 04 04:58:39.772324 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:39.772281 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/373125b2-53a5-449e-9fd5-a79c5fcd06fa-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj\" (UID: \"373125b2-53a5-449e-9fd5-a79c5fcd06fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" Nov 04 04:58:39.772324 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:39.772334 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/373125b2-53a5-449e-9fd5-a79c5fcd06fa-metrics-certs\") pod \"maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj\" (UID: \"373125b2-53a5-449e-9fd5-a79c5fcd06fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" Nov 04 04:58:39.772507 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:39.772356 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/373125b2-53a5-449e-9fd5-a79c5fcd06fa-maas-api-tls\") pod \"maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj\" (UID: \"373125b2-53a5-449e-9fd5-a79c5fcd06fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" Nov 04 04:58:39.772507 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:39.772459 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbnxl\" (UniqueName: \"kubernetes.io/projected/373125b2-53a5-449e-9fd5-a79c5fcd06fa-kube-api-access-jbnxl\") pod \"maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj\" (UID: \"373125b2-53a5-449e-9fd5-a79c5fcd06fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" Nov 04 04:58:39.790182 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:39.790143 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-4cce33-serving-cert\"" Nov 04 04:58:39.793317 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:39.793282 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj"] Nov 04 04:58:39.873754 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:39.873726 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/373125b2-53a5-449e-9fd5-a79c5fcd06fa-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj\" (UID: \"373125b2-53a5-449e-9fd5-a79c5fcd06fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" Nov 04 04:58:39.873946 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:39.873764 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/373125b2-53a5-449e-9fd5-a79c5fcd06fa-metrics-certs\") pod \"maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj\" (UID: \"373125b2-53a5-449e-9fd5-a79c5fcd06fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" Nov 04 04:58:39.873946 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:39.873787 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/373125b2-53a5-449e-9fd5-a79c5fcd06fa-maas-api-tls\") pod \"maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj\" (UID: \"373125b2-53a5-449e-9fd5-a79c5fcd06fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" Nov 04 04:58:39.873946 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:58:39.873887 4581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-serving-cert: secret "maas-api-e2e-worker-w5-4cce33-serving-cert" not found Nov 04 04:58:39.873946 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:58:39.873946 4581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/373125b2-53a5-449e-9fd5-a79c5fcd06fa-maas-api-tls podName:373125b2-53a5-449e-9fd5-a79c5fcd06fa nodeName:}" failed. No retries permitted until 2026-11-04 04:58:40.373928428 +0000 UTC m=+679.264785426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/373125b2-53a5-449e-9fd5-a79c5fcd06fa-maas-api-tls") pod "maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" (UID: "373125b2-53a5-449e-9fd5-a79c5fcd06fa") : secret "maas-api-e2e-worker-w5-4cce33-serving-cert" not found Nov 04 04:58:39.874212 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:39.873975 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbnxl\" (UniqueName: \"kubernetes.io/projected/373125b2-53a5-449e-9fd5-a79c5fcd06fa-kube-api-access-jbnxl\") pod \"maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj\" (UID: \"373125b2-53a5-449e-9fd5-a79c5fcd06fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" Nov 04 04:58:39.874525 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:39.874506 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/373125b2-53a5-449e-9fd5-a79c5fcd06fa-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj\" (UID: \"373125b2-53a5-449e-9fd5-a79c5fcd06fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" Nov 04 04:58:39.876411 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:39.876390 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/373125b2-53a5-449e-9fd5-a79c5fcd06fa-metrics-certs\") pod \"maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj\" (UID: \"373125b2-53a5-449e-9fd5-a79c5fcd06fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" Nov 04 04:58:39.916960 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:39.916934 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:39.931500 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:39.931479 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbnxl\" (UniqueName: \"kubernetes.io/projected/373125b2-53a5-449e-9fd5-a79c5fcd06fa-kube-api-access-jbnxl\") pod \"maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj\" (UID: \"373125b2-53a5-449e-9fd5-a79c5fcd06fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" Nov 04 04:58:40.096515 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:40.096433 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:40.217665 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:40.217630 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:40.379629 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:40.379600 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/373125b2-53a5-449e-9fd5-a79c5fcd06fa-maas-api-tls\") pod \"maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj\" (UID: \"373125b2-53a5-449e-9fd5-a79c5fcd06fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" Nov 04 04:58:40.382167 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:40.382144 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/373125b2-53a5-449e-9fd5-a79c5fcd06fa-maas-api-tls\") pod \"maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj\" (UID: \"373125b2-53a5-449e-9fd5-a79c5fcd06fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" Nov 04 04:58:40.648208 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:40.648123 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" Nov 04 04:58:41.019628 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:41.019601 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj"] Nov 04 04:58:41.021711 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:58:41.021680 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373125b2_53a5_449e_9fd5_a79c5fcd06fa.slice/crio-525dde858f827c7bb7e9db8f8360f1a81d25934064daa87a71ba230f02b2aa64 WatchSource:0}: Error finding container 525dde858f827c7bb7e9db8f8360f1a81d25934064daa87a71ba230f02b2aa64: Status 404 returned error can't find the container with id 525dde858f827c7bb7e9db8f8360f1a81d25934064daa87a71ba230f02b2aa64 Nov 04 04:58:41.502172 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:41.502128 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" event={"ID":"373125b2-53a5-449e-9fd5-a79c5fcd06fa","Type":"ContainerStarted","Data":"525dde858f827c7bb7e9db8f8360f1a81d25934064daa87a71ba230f02b2aa64"} Nov 04 04:58:42.508870 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:42.508830 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" event={"ID":"373125b2-53a5-449e-9fd5-a79c5fcd06fa","Type":"ContainerStarted","Data":"e24108dc7caade34ce24169d49b59eda4c01efc8ed0d2e1d6eb3ecc44a21d50e"} Nov 04 04:58:42.509310 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:42.508968 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" Nov 04 04:58:42.626406 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:42.626352 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" podStartSLOduration=2.809454529 podStartE2EDuration="3.626337905s" totalImagesPullingTime="816.883376ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:58:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:58:42.623949358 +0000 UTC m=+681.514806377" watchObservedRunningTime="2026-11-04 04:58:42.626337905 +0000 UTC m=+681.517194924" Nov 04 04:58:43.476295 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:43.476262 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" Nov 04 04:58:45.199371 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:45.199337 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:45.635174 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:45.635141 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:45.813961 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:45.813923 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:47.886496 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:58:47.886462 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 04:58:48.152637 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:48.152550 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:48.330934 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:48.330896 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:48.519450 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:48.519378 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" Nov 04 04:58:48.717531 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:48.717499 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7"] Nov 04 04:58:48.721849 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:48.721826 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7" Nov 04 04:58:48.769371 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:48.769342 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7b97\" (UniqueName: \"kubernetes.io/projected/4c58df41-54f0-4939-ae70-0e9edb402458-kube-api-access-x7b97\") pod \"payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7\" (UID: \"4c58df41-54f0-4939-ae70-0e9edb402458\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7" Nov 04 04:58:48.769577 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:48.769493 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c58df41-54f0-4939-ae70-0e9edb402458-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7\" (UID: \"4c58df41-54f0-4939-ae70-0e9edb402458\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7" Nov 04 04:58:48.781052 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:48.781021 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7"] Nov 04 04:58:48.783839 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:48.783813 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-1874d8\"" Nov 04 04:58:48.784208 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:48.784161 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-1874d8-dockercfg-54m2q\"" Nov 04 04:58:48.870637 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:48.870600 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c58df41-54f0-4939-ae70-0e9edb402458-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7\" (UID: \"4c58df41-54f0-4939-ae70-0e9edb402458\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7" Nov 04 04:58:48.870791 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:48.870692 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7b97\" (UniqueName: \"kubernetes.io/projected/4c58df41-54f0-4939-ae70-0e9edb402458-kube-api-access-x7b97\") pod \"payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7\" (UID: \"4c58df41-54f0-4939-ae70-0e9edb402458\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7" Nov 04 04:58:48.871351 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:48.871333 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c58df41-54f0-4939-ae70-0e9edb402458-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7\" (UID: \"4c58df41-54f0-4939-ae70-0e9edb402458\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7" Nov 04 04:58:48.927796 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:48.927769 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7b97\" (UniqueName: \"kubernetes.io/projected/4c58df41-54f0-4939-ae70-0e9edb402458-kube-api-access-x7b97\") pod \"payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7\" (UID: \"4c58df41-54f0-4939-ae70-0e9edb402458\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7" Nov 04 04:58:49.035116 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.035040 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7" Nov 04 04:58:49.386604 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.386569 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp"] Nov 04 04:58:49.391617 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.391595 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.409171 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.409146 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7"] Nov 04 04:58:49.410557 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:58:49.410529 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c58df41_54f0_4939_ae70_0e9edb402458.slice/crio-4287d0088315a63955550e2d40458b02bd83e012bfbe610b38a35aeb143c35c0 WatchSource:0}: Error finding container 4287d0088315a63955550e2d40458b02bd83e012bfbe610b38a35aeb143c35c0: Status 404 returned error can't find the container with id 4287d0088315a63955550e2d40458b02bd83e012bfbe610b38a35aeb143c35c0 Nov 04 04:58:49.444500 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.444474 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3eb31c-openshift-default-dockercfg-bscg9\"" Nov 04 04:58:49.478090 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.478055 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-envoy\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.478246 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.478096 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8twsr\" (UniqueName: \"kubernetes.io/projected/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-kube-api-access-8twsr\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.478246 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.478138 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-workload-certs\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.478246 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.478191 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-workload-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.478246 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.478216 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-token\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.478246 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.478242 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-credential-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.478526 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.478269 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-data\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.478526 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.478342 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-podinfo\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.478526 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.478443 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istiod-ca-cert\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.480102 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.480074 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp"] Nov 04 04:58:49.545900 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.545857 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7" event={"ID":"4c58df41-54f0-4939-ae70-0e9edb402458","Type":"ContainerStarted","Data":"c2c4f3660d7224ffe5a20eb59d954baaa05727faf40d99caffc5c7cd760038f0"} Nov 04 04:58:49.545900 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.545906 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7" event={"ID":"4c58df41-54f0-4939-ae70-0e9edb402458","Type":"ContainerStarted","Data":"4287d0088315a63955550e2d40458b02bd83e012bfbe610b38a35aeb143c35c0"} Nov 04 04:58:49.546163 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.546013 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7" Nov 04 04:58:49.579767 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.579742 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-envoy\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.579940 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.579771 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8twsr\" (UniqueName: \"kubernetes.io/projected/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-kube-api-access-8twsr\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.579940 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.579796 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-workload-certs\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.579940 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.579833 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-workload-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.579940 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.579861 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-token\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.579940 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.579887 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-credential-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.579940 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.579910 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-data\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.580296 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.579975 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-podinfo\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.580296 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.580103 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istiod-ca-cert\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.580296 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.580214 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-workload-certs\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.580413 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.580334 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-credential-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.580456 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.580439 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-data\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.580803 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.580783 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-workload-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.580845 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.580831 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istiod-ca-cert\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.582358 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.582338 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-envoy\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.582580 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.582558 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-podinfo\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.637085 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.637028 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-token\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.637199 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.637181 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8twsr\" (UniqueName: \"kubernetes.io/projected/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-kube-api-access-8twsr\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:49.661774 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.661732 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7" podStartSLOduration=1.6617196079999998 podStartE2EDuration="1.661719608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:58:49.659748802 +0000 UTC m=+688.550605821" watchObservedRunningTime="2026-11-04 04:58:49.661719608 +0000 UTC m=+688.552576673" Nov 04 04:58:49.705671 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:49.705643 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:50.132266 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:50.132233 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp"] Nov 04 04:58:50.134835 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:58:50.134806 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ac68d4_0a7a_47aa_afeb_803dfc3f362a.slice/crio-39773d1c9849074c83a0f186a2b1393eb492702e675451fc1b57da4c3e95c012 WatchSource:0}: Error finding container 39773d1c9849074c83a0f186a2b1393eb492702e675451fc1b57da4c3e95c012: Status 404 returned error can't find the container with id 39773d1c9849074c83a0f186a2b1393eb492702e675451fc1b57da4c3e95c012 Nov 04 04:58:50.137410 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:50.137372 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:58:50.137498 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:50.137466 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:58:50.137545 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:50.137495 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:58:50.410514 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:50.410423 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:50.553946 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:50.553906 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" event={"ID":"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a","Type":"ContainerStarted","Data":"3afdef2abe8f0af45f9f4c7ef6532d1be73155eca2267b393dfb509ba0cb7832"} Nov 04 04:58:50.554178 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:50.554034 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" event={"ID":"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a","Type":"ContainerStarted","Data":"39773d1c9849074c83a0f186a2b1393eb492702e675451fc1b57da4c3e95c012"} Nov 04 04:58:50.577574 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:50.577528 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:50.702863 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:50.702764 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:50.706501 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:50.706485 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:50.839980 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:50.839945 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:51.707611 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:51.707580 4581 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 04 04:58:51.708025 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:51.707630 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" podUID="f9ac68d4-0a7a-47aa-afeb-803dfc3f362a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 04 04:58:52.706820 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:52.706782 4581 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 04 04:58:52.706967 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:52.706844 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" podUID="f9ac68d4-0a7a-47aa-afeb-803dfc3f362a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 04 04:58:53.706122 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:53.706092 4581 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 04 04:58:53.706533 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:53.706145 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" podUID="f9ac68d4-0a7a-47aa-afeb-803dfc3f362a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 04 04:58:54.706087 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:54.706054 4581 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 04 04:58:54.706267 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:54.706115 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" podUID="f9ac68d4-0a7a-47aa-afeb-803dfc3f362a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 04 04:58:55.706283 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:55.706245 4581 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 04 04:58:55.706681 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:55.706318 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" podUID="f9ac68d4-0a7a-47aa-afeb-803dfc3f362a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 04 04:58:55.781849 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:55.781813 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:58:56.711164 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:56.711130 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:56.711542 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:56.711462 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:56.712206 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:56.712186 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 04:58:56.879964 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:56.879915 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" podStartSLOduration=7.879901428 podStartE2EDuration="7.879901428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:58:50.68255373 +0000 UTC m=+689.573410751" watchObservedRunningTime="2026-11-04 04:58:56.879901428 +0000 UTC m=+695.770758447" Nov 04 04:58:59.890151 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:58:59.890113 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:00.060461 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:00.060419 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:00.197165 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:00.196717 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:00.364052 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:00.364017 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:00.525379 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:00.525286 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:00.557623 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:00.557598 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7" Nov 04 04:59:00.648309 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:00.648275 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:03.371755 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:03.371720 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:03.991082 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:03.991042 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:04.125345 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:04.125310 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:04.279168 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:04.279081 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:05.295585 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:05.295549 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp"] Nov 04 04:59:05.299563 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:05.299538 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" Nov 04 04:59:05.356289 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:05.356251 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-1874d8-serving-cert\"" Nov 04 04:59:05.397275 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:05.397237 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnr55\" (UniqueName: \"kubernetes.io/projected/6ec52444-399d-4a73-9652-ae7d3f46621d-kube-api-access-hnr55\") pod \"maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp\" (UID: \"6ec52444-399d-4a73-9652-ae7d3f46621d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" Nov 04 04:59:05.397436 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:05.397367 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec52444-399d-4a73-9652-ae7d3f46621d-metrics-certs\") pod \"maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp\" (UID: \"6ec52444-399d-4a73-9652-ae7d3f46621d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" Nov 04 04:59:05.397436 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:05.397413 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ec52444-399d-4a73-9652-ae7d3f46621d-maas-api-tls\") pod \"maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp\" (UID: \"6ec52444-399d-4a73-9652-ae7d3f46621d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" Nov 04 04:59:05.397508 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:05.397458 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec52444-399d-4a73-9652-ae7d3f46621d-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp\" (UID: \"6ec52444-399d-4a73-9652-ae7d3f46621d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" Nov 04 04:59:05.406862 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:05.406830 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp"] Nov 04 04:59:05.489767 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:05.489729 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:05.498847 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:05.498822 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnr55\" (UniqueName: \"kubernetes.io/projected/6ec52444-399d-4a73-9652-ae7d3f46621d-kube-api-access-hnr55\") pod \"maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp\" (UID: \"6ec52444-399d-4a73-9652-ae7d3f46621d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" Nov 04 04:59:05.498942 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:05.498904 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec52444-399d-4a73-9652-ae7d3f46621d-metrics-certs\") pod \"maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp\" (UID: \"6ec52444-399d-4a73-9652-ae7d3f46621d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" Nov 04 04:59:05.498942 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:05.498932 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ec52444-399d-4a73-9652-ae7d3f46621d-maas-api-tls\") pod \"maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp\" (UID: \"6ec52444-399d-4a73-9652-ae7d3f46621d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" Nov 04 04:59:05.499088 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:05.498962 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec52444-399d-4a73-9652-ae7d3f46621d-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp\" (UID: \"6ec52444-399d-4a73-9652-ae7d3f46621d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" Nov 04 04:59:05.499146 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:59:05.499086 4581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-serving-cert: secret "maas-api-e2e-worker-w1-1874d8-serving-cert" not found Nov 04 04:59:05.499195 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:59:05.499164 4581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec52444-399d-4a73-9652-ae7d3f46621d-maas-api-tls podName:6ec52444-399d-4a73-9652-ae7d3f46621d nodeName:}" failed. No retries permitted until 2026-11-04 04:59:05.999144928 +0000 UTC m=+704.890001934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6ec52444-399d-4a73-9652-ae7d3f46621d-maas-api-tls") pod "maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" (UID: "6ec52444-399d-4a73-9652-ae7d3f46621d") : secret "maas-api-e2e-worker-w1-1874d8-serving-cert" not found Nov 04 04:59:05.499737 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:05.499719 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec52444-399d-4a73-9652-ae7d3f46621d-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp\" (UID: \"6ec52444-399d-4a73-9652-ae7d3f46621d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" Nov 04 04:59:05.501547 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:05.501527 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec52444-399d-4a73-9652-ae7d3f46621d-metrics-certs\") pod \"maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp\" (UID: \"6ec52444-399d-4a73-9652-ae7d3f46621d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" Nov 04 04:59:05.581969 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:05.581877 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnr55\" (UniqueName: \"kubernetes.io/projected/6ec52444-399d-4a73-9652-ae7d3f46621d-kube-api-access-hnr55\") pod \"maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp\" (UID: \"6ec52444-399d-4a73-9652-ae7d3f46621d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" Nov 04 04:59:05.638347 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:05.638313 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:05.772345 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:05.758780 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:06.006485 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:06.006439 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ec52444-399d-4a73-9652-ae7d3f46621d-maas-api-tls\") pod \"maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp\" (UID: \"6ec52444-399d-4a73-9652-ae7d3f46621d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" Nov 04 04:59:06.009112 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:06.009087 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ec52444-399d-4a73-9652-ae7d3f46621d-maas-api-tls\") pod \"maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp\" (UID: \"6ec52444-399d-4a73-9652-ae7d3f46621d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" Nov 04 04:59:06.212501 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:06.212461 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" Nov 04 04:59:06.625138 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:06.625094 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp"] Nov 04 04:59:06.627462 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:59:06.627432 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec52444_399d_4a73_9652_ae7d3f46621d.slice/crio-d701e31dfce2a780ae43076ed4383634c5a83d1ada218dccc52801c0e3d4f156 WatchSource:0}: Error finding container d701e31dfce2a780ae43076ed4383634c5a83d1ada218dccc52801c0e3d4f156: Status 404 returned error can't find the container with id d701e31dfce2a780ae43076ed4383634c5a83d1ada218dccc52801c0e3d4f156 Nov 04 04:59:06.695951 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:06.695917 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" event={"ID":"6ec52444-399d-4a73-9652-ae7d3f46621d","Type":"ContainerStarted","Data":"d701e31dfce2a780ae43076ed4383634c5a83d1ada218dccc52801c0e3d4f156"} Nov 04 04:59:07.703099 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:07.703063 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" event={"ID":"6ec52444-399d-4a73-9652-ae7d3f46621d","Type":"ContainerStarted","Data":"1c9cca6d063275d2b734aa1d1d2a51422034f8da5f7794490b705c8c4207ebba"} Nov 04 04:59:07.703549 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:07.703210 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" Nov 04 04:59:07.980056 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:07.980023 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:10.803822 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:10.803776 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:10.977427 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:10.977261 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:11.101645 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:11.101482 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:11.731096 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:11.731063 4581 generic.go:356] "Generic (PLEG): container finished" podID="45926cc8-a8ed-40b8-a5f0-db1336dcc102" containerID="95c9490826687caa0924e3da386f02b17adf3f868cfa41e36fe1a01f0c106350" exitCode=0 Nov 04 04:59:11.731270 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:11.731107 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" event={"ID":"45926cc8-a8ed-40b8-a5f0-db1336dcc102","Type":"ContainerDied","Data":"95c9490826687caa0924e3da386f02b17adf3f868cfa41e36fe1a01f0c106350"} Nov 04 04:59:11.731543 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:11.731525 4581 scope.go:122] "RemoveContainer" containerID="95c9490826687caa0924e3da386f02b17adf3f868cfa41e36fe1a01f0c106350" Nov 04 04:59:11.732917 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:11.732885 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:59:11.733079 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:11.732970 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:59:11.733079 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:11.733045 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:59:11.940184 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:11.940124 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" podStartSLOduration=6.037566916 podStartE2EDuration="6.940109038s" totalImagesPullingTime="902.542122ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:59:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:59:07.84160626 +0000 UTC m=+706.732463280" watchObservedRunningTime="2026-11-04 04:59:11.940109038 +0000 UTC m=+710.830966095" Nov 04 04:59:12.738313 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:12.738280 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" event={"ID":"45926cc8-a8ed-40b8-a5f0-db1336dcc102","Type":"ContainerStarted","Data":"ca55c3fe86220011622c736c4bf21d215f6f3db904b7e564a31f26b0af8b133c"} Nov 04 04:59:13.095545 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:13.095450 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:59:13.096276 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:13.096252 4581 patch_prober.go:29] interesting pod/e2e-worker-w2-370078-openshift-default-584c45775-hp82n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 04:59:13.096440 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:13.096416 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" podUID="45926cc8-a8ed-40b8-a5f0-db1336dcc102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 04:59:13.720771 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:13.720741 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" Nov 04 04:59:14.101974 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:14.101379 4581 patch_prober.go:29] interesting pod/e2e-worker-w2-370078-openshift-default-584c45775-hp82n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 04:59:14.101974 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:14.101468 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" podUID="45926cc8-a8ed-40b8-a5f0-db1336dcc102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 04:59:15.097012 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:15.096447 4581 patch_prober.go:29] interesting pod/e2e-worker-w2-370078-openshift-default-584c45775-hp82n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 04:59:15.097012 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:15.096527 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" podUID="45926cc8-a8ed-40b8-a5f0-db1336dcc102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 04:59:15.831929 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:15.831842 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:16.094617 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:16.094541 4581 patch_prober.go:29] interesting pod/e2e-worker-w2-370078-openshift-default-584c45775-hp82n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 04:59:16.094617 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:16.094597 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" podUID="45926cc8-a8ed-40b8-a5f0-db1336dcc102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 04:59:17.093901 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:17.093863 4581 patch_prober.go:29] interesting pod/e2e-worker-w2-370078-openshift-default-584c45775-hp82n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 04:59:17.094436 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:17.093948 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" podUID="45926cc8-a8ed-40b8-a5f0-db1336dcc102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 04:59:17.898313 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:59:17.898273 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 04:59:18.129911 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:18.129880 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:59:18.130560 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:18.130538 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:59:18.131456 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:18.131428 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 04:59:19.818098 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:19.818062 4581 generic.go:356] "Generic (PLEG): container finished" podID="aa8dcd7f-0f9f-4eef-a831-8d164c92830e" containerID="33c3a7763a73a9cc67e554288a78ac02b3ae5e024361baf5d16f26a0894cc30c" exitCode=0 Nov 04 04:59:19.818649 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:19.818136 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" event={"ID":"aa8dcd7f-0f9f-4eef-a831-8d164c92830e","Type":"ContainerDied","Data":"33c3a7763a73a9cc67e554288a78ac02b3ae5e024361baf5d16f26a0894cc30c"} Nov 04 04:59:19.818649 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:19.818521 4581 scope.go:122] "RemoveContainer" containerID="33c3a7763a73a9cc67e554288a78ac02b3ae5e024361baf5d16f26a0894cc30c" Nov 04 04:59:19.819822 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:19.819793 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:59:19.819883 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:19.819860 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:59:19.819922 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:19.819895 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:59:20.614892 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:20.614860 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:20.826064 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:20.826032 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" event={"ID":"aa8dcd7f-0f9f-4eef-a831-8d164c92830e","Type":"ContainerStarted","Data":"d8df4337b2e39c01778326cb4b841029a4b68f6c1d575c9d5056d087911c0abf"} Nov 04 04:59:21.395236 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:21.395201 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:59:21.395432 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:21.395411 4581 patch_prober.go:29] interesting pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 04 04:59:21.395483 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:21.395457 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" podUID="aa8dcd7f-0f9f-4eef-a831-8d164c92830e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 04 04:59:22.402013 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:22.401819 4581 patch_prober.go:29] interesting pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 04 04:59:22.402013 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:22.401896 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" podUID="aa8dcd7f-0f9f-4eef-a831-8d164c92830e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 04 04:59:23.395474 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:23.395432 4581 patch_prober.go:29] interesting pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 04 04:59:23.395776 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:23.395742 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" podUID="aa8dcd7f-0f9f-4eef-a831-8d164c92830e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 04 04:59:24.021596 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:24.021557 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:24.397015 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:24.396427 4581 patch_prober.go:29] interesting pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 04 04:59:24.397015 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:24.396487 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" podUID="aa8dcd7f-0f9f-4eef-a831-8d164c92830e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 04 04:59:24.689634 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:24.689544 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:24.910081 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:24.910045 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:25.100071 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:25.088543 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:25.402014 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:25.401116 4581 patch_prober.go:29] interesting pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 04 04:59:25.402014 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:25.401209 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" podUID="aa8dcd7f-0f9f-4eef-a831-8d164c92830e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 04 04:59:25.975672 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:25.975640 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:26.399903 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:26.399871 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:59:26.400374 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:26.400217 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:59:26.401019 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:26.400976 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 04:59:29.030397 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:29.030347 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:29.624246 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:29.624212 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:29.763807 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:29.763771 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:29.889224 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:29.889133 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:31.074900 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:31.074860 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:31.911749 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:31.911716 4581 generic.go:356] "Generic (PLEG): container finished" podID="ad5ae17b-aca8-4988-8ed1-7042c62fc32f" containerID="8098505851b03a6fe65a0014515ea965c1f6e89ce749aa093d59401232c0d2dc" exitCode=0 Nov 04 04:59:31.911975 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:31.911785 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" event={"ID":"ad5ae17b-aca8-4988-8ed1-7042c62fc32f","Type":"ContainerDied","Data":"8098505851b03a6fe65a0014515ea965c1f6e89ce749aa093d59401232c0d2dc"} Nov 04 04:59:31.912224 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:31.912208 4581 scope.go:122] "RemoveContainer" containerID="8098505851b03a6fe65a0014515ea965c1f6e89ce749aa093d59401232c0d2dc" Nov 04 04:59:31.913447 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:31.913416 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:59:31.913547 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:31.913485 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:59:31.913547 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:31.913517 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 04:59:32.510971 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:32.510929 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:32.919650 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:32.919619 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" event={"ID":"ad5ae17b-aca8-4988-8ed1-7042c62fc32f","Type":"ContainerStarted","Data":"19e7ca7a1865fad06fc199a041c373515351a514984b767be366b1b52bf20b32"} Nov 04 04:59:32.997961 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:32.997925 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:59:32.999087 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:32.999064 4581 patch_prober.go:29] interesting pod/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 04 04:59:32.999220 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:32.999110 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" podUID="ad5ae17b-aca8-4988-8ed1-7042c62fc32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 04 04:59:33.998389 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:33.998351 4581 patch_prober.go:29] interesting pod/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 04 04:59:33.998806 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:33.998407 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" podUID="ad5ae17b-aca8-4988-8ed1-7042c62fc32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 04 04:59:34.999007 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:34.998957 4581 patch_prober.go:29] interesting pod/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 04 04:59:34.999446 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:34.999053 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" podUID="ad5ae17b-aca8-4988-8ed1-7042c62fc32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 04 04:59:35.998945 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:35.998911 4581 patch_prober.go:29] interesting pod/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 04 04:59:35.999145 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:35.998979 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" podUID="ad5ae17b-aca8-4988-8ed1-7042c62fc32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 04 04:59:36.014222 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:36.014189 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:36.373119 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:36.373087 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:59:36.998379 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:36.998344 4581 patch_prober.go:29] interesting pod/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 04 04:59:36.998559 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:36.998399 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" podUID="ad5ae17b-aca8-4988-8ed1-7042c62fc32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 04 04:59:37.998195 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:37.998155 4581 patch_prober.go:29] interesting pod/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 04 04:59:37.998578 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:37.998227 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" podUID="ad5ae17b-aca8-4988-8ed1-7042c62fc32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 04 04:59:39.002942 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:39.002914 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:59:39.004011 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:39.003977 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 04:59:41.151949 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:41.151913 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:41.667171 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:41.667139 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:42.325924 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:42.325877 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:42.455267 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:42.455232 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:42.592221 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:42.592139 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:43.085997 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:43.085957 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:46.233909 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:46.233870 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:47.886823 ip-10-0-141-132 kubenswrapper[4581]: E1104 04:59:47.886789 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 04:59:48.728564 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:48.728531 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:51.231275 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.231240 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc"] Nov 04 04:59:51.234569 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.234553 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 04:59:51.290648 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.290617 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-4cce33\"/\"kube-root-ca.crt\"" Nov 04 04:59:51.315862 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.315835 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-4cce33\"/\"openshift-service-ca.crt\"" Nov 04 04:59:51.319499 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.319463 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:51.327747 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.327720 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-tmp-dir\") pod \"simulator-w5-4cce33-kserve-6549678c49-vhstc\" (UID: \"44643611-a133-437f-87dd-d5413adb495e\") " pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 04:59:51.327880 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.327759 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrtg2\" (UniqueName: \"kubernetes.io/projected/44643611-a133-437f-87dd-d5413adb495e-kube-api-access-rrtg2\") pod \"simulator-w5-4cce33-kserve-6549678c49-vhstc\" (UID: \"44643611-a133-437f-87dd-d5413adb495e\") " pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 04:59:51.327947 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.327923 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-dshm\") pod \"simulator-w5-4cce33-kserve-6549678c49-vhstc\" (UID: \"44643611-a133-437f-87dd-d5413adb495e\") " pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 04:59:51.328043 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.328021 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44643611-a133-437f-87dd-d5413adb495e-tls-certs\") pod \"simulator-w5-4cce33-kserve-6549678c49-vhstc\" (UID: \"44643611-a133-437f-87dd-d5413adb495e\") " pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 04:59:51.328169 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.328152 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-model-cache\") pod \"simulator-w5-4cce33-kserve-6549678c49-vhstc\" (UID: \"44643611-a133-437f-87dd-d5413adb495e\") " pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 04:59:51.328225 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.328196 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-home\") pod \"simulator-w5-4cce33-kserve-6549678c49-vhstc\" (UID: \"44643611-a133-437f-87dd-d5413adb495e\") " pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 04:59:51.340548 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.340529 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-4cce33\"/\"simulator-w5-4cce33-kserve-self-signed-certs\"" Nov 04 04:59:51.340664 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.340580 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-4cce33\"/\"default-dockercfg-6frkd\"" Nov 04 04:59:51.343258 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.343236 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc"] Nov 04 04:59:51.429633 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.429604 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-home\") pod \"simulator-w5-4cce33-kserve-6549678c49-vhstc\" (UID: \"44643611-a133-437f-87dd-d5413adb495e\") " pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 04:59:51.429810 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.429667 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-tmp-dir\") pod \"simulator-w5-4cce33-kserve-6549678c49-vhstc\" (UID: \"44643611-a133-437f-87dd-d5413adb495e\") " pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 04:59:51.429810 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.429692 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrtg2\" (UniqueName: \"kubernetes.io/projected/44643611-a133-437f-87dd-d5413adb495e-kube-api-access-rrtg2\") pod \"simulator-w5-4cce33-kserve-6549678c49-vhstc\" (UID: \"44643611-a133-437f-87dd-d5413adb495e\") " pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 04:59:51.429810 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.429750 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-dshm\") pod \"simulator-w5-4cce33-kserve-6549678c49-vhstc\" (UID: \"44643611-a133-437f-87dd-d5413adb495e\") " pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 04:59:51.429810 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.429796 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44643611-a133-437f-87dd-d5413adb495e-tls-certs\") pod \"simulator-w5-4cce33-kserve-6549678c49-vhstc\" (UID: \"44643611-a133-437f-87dd-d5413adb495e\") " pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 04:59:51.430025 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.429879 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-model-cache\") pod \"simulator-w5-4cce33-kserve-6549678c49-vhstc\" (UID: \"44643611-a133-437f-87dd-d5413adb495e\") " pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 04:59:51.430108 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.430079 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-home\") pod \"simulator-w5-4cce33-kserve-6549678c49-vhstc\" (UID: \"44643611-a133-437f-87dd-d5413adb495e\") " pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 04:59:51.430283 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.430261 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-model-cache\") pod \"simulator-w5-4cce33-kserve-6549678c49-vhstc\" (UID: \"44643611-a133-437f-87dd-d5413adb495e\") " pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 04:59:51.430369 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.430355 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-tmp-dir\") pod \"simulator-w5-4cce33-kserve-6549678c49-vhstc\" (UID: \"44643611-a133-437f-87dd-d5413adb495e\") " pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 04:59:51.432237 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.432215 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-dshm\") pod \"simulator-w5-4cce33-kserve-6549678c49-vhstc\" (UID: \"44643611-a133-437f-87dd-d5413adb495e\") " pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 04:59:51.432393 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.432374 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44643611-a133-437f-87dd-d5413adb495e-tls-certs\") pod \"simulator-w5-4cce33-kserve-6549678c49-vhstc\" (UID: \"44643611-a133-437f-87dd-d5413adb495e\") " pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 04:59:51.487899 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.487834 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrtg2\" (UniqueName: \"kubernetes.io/projected/44643611-a133-437f-87dd-d5413adb495e-kube-api-access-rrtg2\") pod \"simulator-w5-4cce33-kserve-6549678c49-vhstc\" (UID: \"44643611-a133-437f-87dd-d5413adb495e\") " pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 04:59:51.490336 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.490299 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:51.545696 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.545666 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 04:59:51.917239 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:51.917215 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc"] Nov 04 04:59:51.919556 ip-10-0-141-132 kubenswrapper[4581]: W1104 04:59:51.919524 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44643611_a133_437f_87dd_d5413adb495e.slice/crio-91f36421ff0abc28abbf134dd82abb68488eb33562b86ab4068b90a98ddb9ad0 WatchSource:0}: Error finding container 91f36421ff0abc28abbf134dd82abb68488eb33562b86ab4068b90a98ddb9ad0: Status 404 returned error can't find the container with id 91f36421ff0abc28abbf134dd82abb68488eb33562b86ab4068b90a98ddb9ad0 Nov 04 04:59:52.024924 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:52.024889 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" event={"ID":"44643611-a133-437f-87dd-d5413adb495e","Type":"ContainerStarted","Data":"09cbe4b827fab8a6effbc45f762d8cb373e01cff7e983b6a68fecdf953e7f46b"} Nov 04 04:59:52.025145 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:52.024939 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" event={"ID":"44643611-a133-437f-87dd-d5413adb495e","Type":"ContainerStarted","Data":"91f36421ff0abc28abbf134dd82abb68488eb33562b86ab4068b90a98ddb9ad0"} Nov 04 04:59:52.141496 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:52.141439 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" podStartSLOduration=1.141422287 podStartE2EDuration="1.141422287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:59:52.140007623 +0000 UTC m=+751.030864635" watchObservedRunningTime="2026-11-04 04:59:52.141422287 +0000 UTC m=+751.032279308" Nov 04 04:59:52.546314 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:52.546285 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 04:59:54.924896 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:54.924858 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:55.555230 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:55.555187 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:55.694789 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:55.694755 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:55.830344 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:55.830244 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:56.432797 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:56.432761 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:59.381175 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:59.381137 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:59.526290 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:59.526255 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 04:59:59.664292 ip-10-0-141-132 kubenswrapper[4581]: I1104 04:59:59.664192 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:00.236726 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:00.236684 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:00.424972 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:00.424924 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:00.682868 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:00.682838 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:01.322672 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:01.322637 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:01.515763 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:01.515725 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:01.545864 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:01.545832 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 05:00:01.559655 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:01.559629 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 05:00:01.686933 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:01.686881 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:02.097179 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:02.097068 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 05:00:04.493853 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.493818 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb"] Nov 04 05:00:04.496751 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.496732 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:00:04.569466 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.569432 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb"] Nov 04 05:00:04.569757 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.569733 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:04.574164 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.574141 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsvvd\" (UniqueName: \"kubernetes.io/projected/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-kube-api-access-wsvvd\") pod \"simulator-w2-370078-kserve-5b84dbd9f4-c8bjb\" (UID: \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\") " pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:00:04.574253 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.574210 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-model-cache\") pod \"simulator-w2-370078-kserve-5b84dbd9f4-c8bjb\" (UID: \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\") " pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:00:04.574332 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.574313 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-home\") pod \"simulator-w2-370078-kserve-5b84dbd9f4-c8bjb\" (UID: \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\") " pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:00:04.574380 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.574367 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-tmp-dir\") pod \"simulator-w2-370078-kserve-5b84dbd9f4-c8bjb\" (UID: \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\") " pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:00:04.574475 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.574455 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-dshm\") pod \"simulator-w2-370078-kserve-5b84dbd9f4-c8bjb\" (UID: \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\") " pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:00:04.574519 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.574505 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-tls-certs\") pod \"simulator-w2-370078-kserve-5b84dbd9f4-c8bjb\" (UID: \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\") " pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:00:04.602597 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.602559 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-370078\"/\"openshift-service-ca.crt\"" Nov 04 05:00:04.602728 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.602601 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-370078\"/\"kube-root-ca.crt\"" Nov 04 05:00:04.602728 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.602673 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-370078\"/\"default-dockercfg-ksfsj\"" Nov 04 05:00:04.602809 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.602753 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-370078\"/\"simulator-w2-370078-kserve-self-signed-certs\"" Nov 04 05:00:04.675753 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.675717 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-model-cache\") pod \"simulator-w2-370078-kserve-5b84dbd9f4-c8bjb\" (UID: \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\") " pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:00:04.675962 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.675797 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-home\") pod \"simulator-w2-370078-kserve-5b84dbd9f4-c8bjb\" (UID: \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\") " pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:00:04.675962 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.675851 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-tmp-dir\") pod \"simulator-w2-370078-kserve-5b84dbd9f4-c8bjb\" (UID: \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\") " pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:00:04.675962 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.675899 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-dshm\") pod \"simulator-w2-370078-kserve-5b84dbd9f4-c8bjb\" (UID: \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\") " pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:00:04.675962 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.675949 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-tls-certs\") pod \"simulator-w2-370078-kserve-5b84dbd9f4-c8bjb\" (UID: \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\") " pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:00:04.676202 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.675978 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsvvd\" (UniqueName: \"kubernetes.io/projected/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-kube-api-access-wsvvd\") pod \"simulator-w2-370078-kserve-5b84dbd9f4-c8bjb\" (UID: \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\") " pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:00:04.676260 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.676223 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-model-cache\") pod \"simulator-w2-370078-kserve-5b84dbd9f4-c8bjb\" (UID: \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\") " pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:00:04.676310 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.676255 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-home\") pod \"simulator-w2-370078-kserve-5b84dbd9f4-c8bjb\" (UID: \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\") " pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:00:04.676310 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.676288 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-tmp-dir\") pod \"simulator-w2-370078-kserve-5b84dbd9f4-c8bjb\" (UID: \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\") " pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:00:04.678411 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.678384 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-dshm\") pod \"simulator-w2-370078-kserve-5b84dbd9f4-c8bjb\" (UID: \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\") " pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:00:04.678544 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.678526 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-tls-certs\") pod \"simulator-w2-370078-kserve-5b84dbd9f4-c8bjb\" (UID: \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\") " pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:00:04.733657 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.733626 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsvvd\" (UniqueName: \"kubernetes.io/projected/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-kube-api-access-wsvvd\") pod \"simulator-w2-370078-kserve-5b84dbd9f4-c8bjb\" (UID: \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\") " pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:00:04.808661 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:04.808575 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:00:05.182976 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:05.182947 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb"] Nov 04 05:00:05.184695 ip-10-0-141-132 kubenswrapper[4581]: W1104 05:00:05.184658 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a196cc6_b6bd_4fb1_8fb7_9588bb4ff674.slice/crio-002e88e8071a35a8201cd235316ba46eb8bbc1707640d4301bda276824f21bd3 WatchSource:0}: Error finding container 002e88e8071a35a8201cd235316ba46eb8bbc1707640d4301bda276824f21bd3: Status 404 returned error can't find the container with id 002e88e8071a35a8201cd235316ba46eb8bbc1707640d4301bda276824f21bd3 Nov 04 05:00:06.111854 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:06.111811 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" event={"ID":"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674","Type":"ContainerStarted","Data":"a7ccc60144f0e802d0949d8358f006a12a39dcfa0dcbc8099fae005653d0bce5"} Nov 04 05:00:06.112312 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:06.111865 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" event={"ID":"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674","Type":"ContainerStarted","Data":"002e88e8071a35a8201cd235316ba46eb8bbc1707640d4301bda276824f21bd3"} Nov 04 05:00:06.230044 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:06.229967 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" podStartSLOduration=2.229948656 podStartE2EDuration="2.229948656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:00:06.227013691 +0000 UTC m=+765.117870716" watchObservedRunningTime="2026-11-04 05:00:06.229948656 +0000 UTC m=+765.120805679" Nov 04 05:00:06.264781 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:06.264748 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:06.808995 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:06.808963 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:00:11.057636 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:11.057588 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:14.809682 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:14.809648 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:00:14.822619 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:14.822595 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:00:15.176256 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:15.176231 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:00:15.824731 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:15.824649 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:17.886505 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:00:17.886462 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 05:00:20.531157 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:20.531124 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:21.658490 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:21.658454 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:25.335776 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:25.335740 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:30.031326 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:30.031297 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:34.736436 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:34.736396 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:35.714428 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:35.714392 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:37.719774 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:37.719739 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:39.526248 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:39.526214 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:41.482918 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:41.482885 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:44.327033 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:44.326981 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:47.889656 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:00:47.889610 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 05:00:48.669171 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.669140 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr"] Nov 04 05:00:48.672262 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.672245 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:00:48.756679 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.756642 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:48.772636 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.772606 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-dshm\") pod \"premium-w6-cedfca-kserve-7b859ff974-fj8mr\" (UID: \"9414f040-9632-4245-b1c3-b00b229cb75a\") " pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:00:48.772756 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.772705 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-tmp-dir\") pod \"premium-w6-cedfca-kserve-7b859ff974-fj8mr\" (UID: \"9414f040-9632-4245-b1c3-b00b229cb75a\") " pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:00:48.772816 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.772758 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-home\") pod \"premium-w6-cedfca-kserve-7b859ff974-fj8mr\" (UID: \"9414f040-9632-4245-b1c3-b00b229cb75a\") " pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:00:48.772816 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.772781 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psx6w\" (UniqueName: \"kubernetes.io/projected/9414f040-9632-4245-b1c3-b00b229cb75a-kube-api-access-psx6w\") pod \"premium-w6-cedfca-kserve-7b859ff974-fj8mr\" (UID: \"9414f040-9632-4245-b1c3-b00b229cb75a\") " pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:00:48.772816 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.772800 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9414f040-9632-4245-b1c3-b00b229cb75a-tls-certs\") pod \"premium-w6-cedfca-kserve-7b859ff974-fj8mr\" (UID: \"9414f040-9632-4245-b1c3-b00b229cb75a\") " pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:00:48.772975 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.772940 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-model-cache\") pod \"premium-w6-cedfca-kserve-7b859ff974-fj8mr\" (UID: \"9414f040-9632-4245-b1c3-b00b229cb75a\") " pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:00:48.778448 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.778422 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-cedfca\"/\"premium-w6-cedfca-kserve-self-signed-certs\"" Nov 04 05:00:48.778561 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.778453 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-cedfca\"/\"kube-root-ca.crt\"" Nov 04 05:00:48.778561 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.778487 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-cedfca\"/\"default-dockercfg-f7vjm\"" Nov 04 05:00:48.778561 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.778540 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-cedfca\"/\"openshift-service-ca.crt\"" Nov 04 05:00:48.781013 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.780975 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr"] Nov 04 05:00:48.874195 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.874168 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-dshm\") pod \"premium-w6-cedfca-kserve-7b859ff974-fj8mr\" (UID: \"9414f040-9632-4245-b1c3-b00b229cb75a\") " pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:00:48.874374 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.874242 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-tmp-dir\") pod \"premium-w6-cedfca-kserve-7b859ff974-fj8mr\" (UID: \"9414f040-9632-4245-b1c3-b00b229cb75a\") " pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:00:48.874374 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.874273 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-home\") pod \"premium-w6-cedfca-kserve-7b859ff974-fj8mr\" (UID: \"9414f040-9632-4245-b1c3-b00b229cb75a\") " pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:00:48.874374 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.874302 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psx6w\" (UniqueName: \"kubernetes.io/projected/9414f040-9632-4245-b1c3-b00b229cb75a-kube-api-access-psx6w\") pod \"premium-w6-cedfca-kserve-7b859ff974-fj8mr\" (UID: \"9414f040-9632-4245-b1c3-b00b229cb75a\") " pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:00:48.874374 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.874334 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9414f040-9632-4245-b1c3-b00b229cb75a-tls-certs\") pod \"premium-w6-cedfca-kserve-7b859ff974-fj8mr\" (UID: \"9414f040-9632-4245-b1c3-b00b229cb75a\") " pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:00:48.874612 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.874393 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-model-cache\") pod \"premium-w6-cedfca-kserve-7b859ff974-fj8mr\" (UID: \"9414f040-9632-4245-b1c3-b00b229cb75a\") " pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:00:48.874744 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.874699 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-home\") pod \"premium-w6-cedfca-kserve-7b859ff974-fj8mr\" (UID: \"9414f040-9632-4245-b1c3-b00b229cb75a\") " pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:00:48.875061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.874748 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-model-cache\") pod \"premium-w6-cedfca-kserve-7b859ff974-fj8mr\" (UID: \"9414f040-9632-4245-b1c3-b00b229cb75a\") " pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:00:48.875061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.874919 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-tmp-dir\") pod \"premium-w6-cedfca-kserve-7b859ff974-fj8mr\" (UID: \"9414f040-9632-4245-b1c3-b00b229cb75a\") " pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:00:48.876526 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.876504 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-dshm\") pod \"premium-w6-cedfca-kserve-7b859ff974-fj8mr\" (UID: \"9414f040-9632-4245-b1c3-b00b229cb75a\") " pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:00:48.877118 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.877099 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9414f040-9632-4245-b1c3-b00b229cb75a-tls-certs\") pod \"premium-w6-cedfca-kserve-7b859ff974-fj8mr\" (UID: \"9414f040-9632-4245-b1c3-b00b229cb75a\") " pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:00:48.934400 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.934338 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psx6w\" (UniqueName: \"kubernetes.io/projected/9414f040-9632-4245-b1c3-b00b229cb75a-kube-api-access-psx6w\") pod \"premium-w6-cedfca-kserve-7b859ff974-fj8mr\" (UID: \"9414f040-9632-4245-b1c3-b00b229cb75a\") " pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:00:48.983152 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:48.983120 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:00:49.129690 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:49.129653 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:49.360834 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:49.360810 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr"] Nov 04 05:00:49.362937 ip-10-0-141-132 kubenswrapper[4581]: W1104 05:00:49.362906 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9414f040_9632_4245_b1c3_b00b229cb75a.slice/crio-1aedc8657e60150f8a3797fb8860ab690f5e2258157006e6148084edca23f21a WatchSource:0}: Error finding container 1aedc8657e60150f8a3797fb8860ab690f5e2258157006e6148084edca23f21a: Status 404 returned error can't find the container with id 1aedc8657e60150f8a3797fb8860ab690f5e2258157006e6148084edca23f21a Nov 04 05:00:50.367847 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:50.367798 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" event={"ID":"9414f040-9632-4245-b1c3-b00b229cb75a","Type":"ContainerStarted","Data":"18504455d82ab778be4b6847592cac5c263006b128509002f016a5a8a4676d4d"} Nov 04 05:00:50.367847 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:50.367852 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" event={"ID":"9414f040-9632-4245-b1c3-b00b229cb75a","Type":"ContainerStarted","Data":"1aedc8657e60150f8a3797fb8860ab690f5e2258157006e6148084edca23f21a"} Nov 04 05:00:50.485903 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:50.485851 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" podStartSLOduration=2.485832612 podStartE2EDuration="2.485832612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:00:50.483804726 +0000 UTC m=+809.374661759" watchObservedRunningTime="2026-11-04 05:00:50.485832612 +0000 UTC m=+809.376689635" Nov 04 05:00:50.983713 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:50.983682 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:00:53.854677 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:53.854641 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:57.476510 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:57.476475 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:58.533373 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:58.533338 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:00:58.984177 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:58.984145 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:00:58.996815 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:58.996793 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:00:59.425897 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:00:59.425867 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:01:00.733582 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:00.733544 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:01:03.227276 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:03.227235 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:01:07.966593 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:07.966548 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:01:12.754628 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:12.754583 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:01:17.514709 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:17.514678 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:01:17.887166 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:01:17.887127 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 05:01:22.280811 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:22.280775 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:01:26.345877 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:26.345843 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:01:27.143511 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:27.143472 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:01:31.931309 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:31.931273 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:01:36.642340 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:36.642299 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:01:41.439555 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:41.439519 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:01:41.899640 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:41.899613 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2"] Nov 04 05:01:41.903372 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:41.903356 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:01:41.962154 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:41.962123 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-home\") pod \"premium-w1-1874d8-kserve-586794cbc9-vvht2\" (UID: \"1c7c486a-863f-49ff-abb3-819e3905e640\") " pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:01:41.962350 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:41.962176 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-tmp-dir\") pod \"premium-w1-1874d8-kserve-586794cbc9-vvht2\" (UID: \"1c7c486a-863f-49ff-abb3-819e3905e640\") " pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:01:41.962350 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:41.962333 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-dshm\") pod \"premium-w1-1874d8-kserve-586794cbc9-vvht2\" (UID: \"1c7c486a-863f-49ff-abb3-819e3905e640\") " pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:01:41.962445 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:41.962404 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c7c486a-863f-49ff-abb3-819e3905e640-tls-certs\") pod \"premium-w1-1874d8-kserve-586794cbc9-vvht2\" (UID: \"1c7c486a-863f-49ff-abb3-819e3905e640\") " pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:01:41.962512 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:41.962498 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqdfv\" (UniqueName: \"kubernetes.io/projected/1c7c486a-863f-49ff-abb3-819e3905e640-kube-api-access-dqdfv\") pod \"premium-w1-1874d8-kserve-586794cbc9-vvht2\" (UID: \"1c7c486a-863f-49ff-abb3-819e3905e640\") " pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:01:41.962614 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:41.962600 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-model-cache\") pod \"premium-w1-1874d8-kserve-586794cbc9-vvht2\" (UID: \"1c7c486a-863f-49ff-abb3-819e3905e640\") " pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:01:41.974249 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:41.974221 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2"] Nov 04 05:01:41.978408 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:41.978375 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:01:42.007616 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:42.007593 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1874d8\"/\"default-dockercfg-h2jcg\"" Nov 04 05:01:42.007616 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:42.007597 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1874d8\"/\"openshift-service-ca.crt\"" Nov 04 05:01:42.007824 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:42.007651 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1874d8\"/\"kube-root-ca.crt\"" Nov 04 05:01:42.007824 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:42.007658 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1874d8\"/\"premium-w1-1874d8-kserve-self-signed-certs\"" Nov 04 05:01:42.064266 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:42.064237 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-model-cache\") pod \"premium-w1-1874d8-kserve-586794cbc9-vvht2\" (UID: \"1c7c486a-863f-49ff-abb3-819e3905e640\") " pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:01:42.064457 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:42.064277 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-home\") pod \"premium-w1-1874d8-kserve-586794cbc9-vvht2\" (UID: \"1c7c486a-863f-49ff-abb3-819e3905e640\") " pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:01:42.064457 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:42.064302 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-tmp-dir\") pod \"premium-w1-1874d8-kserve-586794cbc9-vvht2\" (UID: \"1c7c486a-863f-49ff-abb3-819e3905e640\") " pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:01:42.064457 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:42.064353 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-dshm\") pod \"premium-w1-1874d8-kserve-586794cbc9-vvht2\" (UID: \"1c7c486a-863f-49ff-abb3-819e3905e640\") " pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:01:42.064457 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:42.064396 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c7c486a-863f-49ff-abb3-819e3905e640-tls-certs\") pod \"premium-w1-1874d8-kserve-586794cbc9-vvht2\" (UID: \"1c7c486a-863f-49ff-abb3-819e3905e640\") " pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:01:42.064457 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:42.064444 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqdfv\" (UniqueName: \"kubernetes.io/projected/1c7c486a-863f-49ff-abb3-819e3905e640-kube-api-access-dqdfv\") pod \"premium-w1-1874d8-kserve-586794cbc9-vvht2\" (UID: \"1c7c486a-863f-49ff-abb3-819e3905e640\") " pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:01:42.064707 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:42.064678 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-home\") pod \"premium-w1-1874d8-kserve-586794cbc9-vvht2\" (UID: \"1c7c486a-863f-49ff-abb3-819e3905e640\") " pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:01:42.064767 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:42.064730 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-model-cache\") pod \"premium-w1-1874d8-kserve-586794cbc9-vvht2\" (UID: \"1c7c486a-863f-49ff-abb3-819e3905e640\") " pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:01:42.064824 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:42.064773 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-tmp-dir\") pod \"premium-w1-1874d8-kserve-586794cbc9-vvht2\" (UID: \"1c7c486a-863f-49ff-abb3-819e3905e640\") " pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:01:42.066884 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:42.066857 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-dshm\") pod \"premium-w1-1874d8-kserve-586794cbc9-vvht2\" (UID: \"1c7c486a-863f-49ff-abb3-819e3905e640\") " pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:01:42.067023 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:42.067002 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c7c486a-863f-49ff-abb3-819e3905e640-tls-certs\") pod \"premium-w1-1874d8-kserve-586794cbc9-vvht2\" (UID: \"1c7c486a-863f-49ff-abb3-819e3905e640\") " pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:01:42.122880 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:42.122856 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqdfv\" (UniqueName: \"kubernetes.io/projected/1c7c486a-863f-49ff-abb3-819e3905e640-kube-api-access-dqdfv\") pod \"premium-w1-1874d8-kserve-586794cbc9-vvht2\" (UID: \"1c7c486a-863f-49ff-abb3-819e3905e640\") " pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:01:42.214969 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:42.214887 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:01:42.610018 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:42.609964 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2"] Nov 04 05:01:42.614968 ip-10-0-141-132 kubenswrapper[4581]: W1104 05:01:42.614940 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c7c486a_863f_49ff_abb3_819e3905e640.slice/crio-0ce6616246c72a56d74fa53b93090b1f089127aa8df0bf16c8fec7f82c6f0dc1 WatchSource:0}: Error finding container 0ce6616246c72a56d74fa53b93090b1f089127aa8df0bf16c8fec7f82c6f0dc1: Status 404 returned error can't find the container with id 0ce6616246c72a56d74fa53b93090b1f089127aa8df0bf16c8fec7f82c6f0dc1 Nov 04 05:01:42.663681 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:42.663651 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" event={"ID":"1c7c486a-863f-49ff-abb3-819e3905e640","Type":"ContainerStarted","Data":"0ce6616246c72a56d74fa53b93090b1f089127aa8df0bf16c8fec7f82c6f0dc1"} Nov 04 05:01:43.672118 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:43.672083 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" event={"ID":"1c7c486a-863f-49ff-abb3-819e3905e640","Type":"ContainerStarted","Data":"2c450522cd19ab5f613fc4d66a9d9cfe290dc9f833a892624e1dc735e70571fc"} Nov 04 05:01:43.792149 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:43.792099 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" podStartSLOduration=2.792086482 podStartE2EDuration="2.792086482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:01:43.789451042 +0000 UTC m=+862.680308063" watchObservedRunningTime="2026-11-04 05:01:43.792086482 +0000 UTC m=+862.682943501" Nov 04 05:01:44.215131 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:44.215102 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:01:46.255033 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:46.254976 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:01:47.664293 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:47.664260 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:01:47.887816 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:01:47.887771 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 05:01:51.141937 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:51.141883 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:01:52.216000 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:52.215962 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:01:52.228856 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:52.228831 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:01:52.737613 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:52.737582 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:01:55.942808 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:55.942774 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:01:56.587886 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.587853 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m"] Nov 04 05:01:56.591836 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.591819 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:01:56.632330 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.632302 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-home\") pod \"premium-w0-0aae2f-kserve-5bcd79c557-xss8m\" (UID: \"ec7b6b61-568d-410e-80fb-c51216be67e3\") " pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:01:56.632457 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.632348 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgc4h\" (UniqueName: \"kubernetes.io/projected/ec7b6b61-568d-410e-80fb-c51216be67e3-kube-api-access-kgc4h\") pod \"premium-w0-0aae2f-kserve-5bcd79c557-xss8m\" (UID: \"ec7b6b61-568d-410e-80fb-c51216be67e3\") " pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:01:56.632503 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.632481 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-dshm\") pod \"premium-w0-0aae2f-kserve-5bcd79c557-xss8m\" (UID: \"ec7b6b61-568d-410e-80fb-c51216be67e3\") " pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:01:56.632538 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.632521 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-model-cache\") pod \"premium-w0-0aae2f-kserve-5bcd79c557-xss8m\" (UID: \"ec7b6b61-568d-410e-80fb-c51216be67e3\") " pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:01:56.632584 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.632572 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-tmp-dir\") pod \"premium-w0-0aae2f-kserve-5bcd79c557-xss8m\" (UID: \"ec7b6b61-568d-410e-80fb-c51216be67e3\") " pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:01:56.632628 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.632617 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec7b6b61-568d-410e-80fb-c51216be67e3-tls-certs\") pod \"premium-w0-0aae2f-kserve-5bcd79c557-xss8m\" (UID: \"ec7b6b61-568d-410e-80fb-c51216be67e3\") " pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:01:56.652799 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.652766 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0aae2f\"/\"kube-root-ca.crt\"" Nov 04 05:01:56.657371 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.657349 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0aae2f\"/\"openshift-service-ca.crt\"" Nov 04 05:01:56.706189 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.706161 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m"] Nov 04 05:01:56.708244 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.708221 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0aae2f\"/\"default-dockercfg-zwkcg\"" Nov 04 05:01:56.708386 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.708270 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0aae2f\"/\"premium-w0-0aae2f-kserve-self-signed-certs\"" Nov 04 05:01:56.734133 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.734107 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-dshm\") pod \"premium-w0-0aae2f-kserve-5bcd79c557-xss8m\" (UID: \"ec7b6b61-568d-410e-80fb-c51216be67e3\") " pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:01:56.734265 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.734140 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-model-cache\") pod \"premium-w0-0aae2f-kserve-5bcd79c557-xss8m\" (UID: \"ec7b6b61-568d-410e-80fb-c51216be67e3\") " pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:01:56.734265 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.734159 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-tmp-dir\") pod \"premium-w0-0aae2f-kserve-5bcd79c557-xss8m\" (UID: \"ec7b6b61-568d-410e-80fb-c51216be67e3\") " pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:01:56.734265 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.734195 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec7b6b61-568d-410e-80fb-c51216be67e3-tls-certs\") pod \"premium-w0-0aae2f-kserve-5bcd79c557-xss8m\" (UID: \"ec7b6b61-568d-410e-80fb-c51216be67e3\") " pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:01:56.734265 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.734256 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-home\") pod \"premium-w0-0aae2f-kserve-5bcd79c557-xss8m\" (UID: \"ec7b6b61-568d-410e-80fb-c51216be67e3\") " pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:01:56.734472 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.734306 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgc4h\" (UniqueName: \"kubernetes.io/projected/ec7b6b61-568d-410e-80fb-c51216be67e3-kube-api-access-kgc4h\") pod \"premium-w0-0aae2f-kserve-5bcd79c557-xss8m\" (UID: \"ec7b6b61-568d-410e-80fb-c51216be67e3\") " pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:01:56.734634 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.734607 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-model-cache\") pod \"premium-w0-0aae2f-kserve-5bcd79c557-xss8m\" (UID: \"ec7b6b61-568d-410e-80fb-c51216be67e3\") " pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:01:56.734781 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.734622 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-tmp-dir\") pod \"premium-w0-0aae2f-kserve-5bcd79c557-xss8m\" (UID: \"ec7b6b61-568d-410e-80fb-c51216be67e3\") " pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:01:56.734781 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.734683 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-home\") pod \"premium-w0-0aae2f-kserve-5bcd79c557-xss8m\" (UID: \"ec7b6b61-568d-410e-80fb-c51216be67e3\") " pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:01:56.736494 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.736472 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-dshm\") pod \"premium-w0-0aae2f-kserve-5bcd79c557-xss8m\" (UID: \"ec7b6b61-568d-410e-80fb-c51216be67e3\") " pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:01:56.736798 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.736779 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec7b6b61-568d-410e-80fb-c51216be67e3-tls-certs\") pod \"premium-w0-0aae2f-kserve-5bcd79c557-xss8m\" (UID: \"ec7b6b61-568d-410e-80fb-c51216be67e3\") " pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:01:56.741029 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.740978 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:01:56.791655 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.791607 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgc4h\" (UniqueName: \"kubernetes.io/projected/ec7b6b61-568d-410e-80fb-c51216be67e3-kube-api-access-kgc4h\") pod \"premium-w0-0aae2f-kserve-5bcd79c557-xss8m\" (UID: \"ec7b6b61-568d-410e-80fb-c51216be67e3\") " pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:01:56.904852 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:56.904785 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:01:57.279606 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:57.279581 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m"] Nov 04 05:01:57.281035 ip-10-0-141-132 kubenswrapper[4581]: W1104 05:01:57.281009 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec7b6b61_568d_410e_80fb_c51216be67e3.slice/crio-424f76a097e3cda3b74714744f51726460d4f0edafbb552d8d97123d342d371d WatchSource:0}: Error finding container 424f76a097e3cda3b74714744f51726460d4f0edafbb552d8d97123d342d371d: Status 404 returned error can't find the container with id 424f76a097e3cda3b74714744f51726460d4f0edafbb552d8d97123d342d371d Nov 04 05:01:57.758868 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:57.758836 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" event={"ID":"ec7b6b61-568d-410e-80fb-c51216be67e3","Type":"ContainerStarted","Data":"75afb13314116ceeabf8b62fedc915eeb169d8aa8084092cd0a0f836d1d8419e"} Nov 04 05:01:57.759099 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:57.758879 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" event={"ID":"ec7b6b61-568d-410e-80fb-c51216be67e3","Type":"ContainerStarted","Data":"424f76a097e3cda3b74714744f51726460d4f0edafbb552d8d97123d342d371d"} Nov 04 05:01:57.875519 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:57.875463 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" podStartSLOduration=1.875447387 podStartE2EDuration="1.875447387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:01:57.874599495 +0000 UTC m=+876.765456514" watchObservedRunningTime="2026-11-04 05:01:57.875447387 +0000 UTC m=+876.766304409" Nov 04 05:01:57.905773 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:01:57.905743 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:02:00.811031 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:00.810997 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:02:01.088386 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:01.088280 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:02:05.631801 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:05.631766 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:02:06.905637 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:06.905608 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:02:06.918096 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:06.918072 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:02:07.824372 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:07.824343 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:02:10.420801 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:10.420756 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:02:15.140284 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:15.140245 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:02:17.888299 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:02:17.888256 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 05:02:19.944044 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:19.943980 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:02:22.716523 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:22.716489 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:02:24.711764 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:24.711726 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:02:29.435014 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:29.434948 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:02:32.180352 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:32.180318 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7"] Nov 04 05:02:32.180829 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:32.180570 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" podUID="2079faa0-1dfe-43f2-8f3a-199e92ff79e7" containerName="istio-proxy" containerID="cri-o://8ce9aa27131079b1f4048ccbf4478c21c5fa6084d56f025ffa6c3291646d2813" gracePeriod=30 Nov 04 05:02:34.329152 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:34.329105 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:02:37.443853 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.443793 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 05:02:37.503585 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.503558 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-podinfo\") pod \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " Nov 04 05:02:37.503727 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.503605 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istiod-ca-cert\") pod \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " Nov 04 05:02:37.503727 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.503643 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-workload-certs\") pod \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " Nov 04 05:02:37.503727 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.503713 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-data\") pod \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " Nov 04 05:02:37.503891 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.503732 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-workload-socket\") pod \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " Nov 04 05:02:37.503891 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.503757 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-token\" (UniqueName: \"kubernetes.io/projected/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-token\") pod \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " Nov 04 05:02:37.503891 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.503776 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-kube-api-access-cgggv\" (UniqueName: \"kubernetes.io/projected/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-kube-api-access-cgggv\") pod \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " Nov 04 05:02:37.503891 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.503849 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-credential-socket\") pod \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " Nov 04 05:02:37.504116 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.503911 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-envoy\") pod \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\" (UID: \"2079faa0-1dfe-43f2-8f3a-199e92ff79e7\") " Nov 04 05:02:37.504116 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.504046 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-data" pod "2079faa0-1dfe-43f2-8f3a-199e92ff79e7" (UID: "2079faa0-1dfe-43f2-8f3a-199e92ff79e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:37.504116 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.504062 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-workload-socket" pod "2079faa0-1dfe-43f2-8f3a-199e92ff79e7" (UID: "2079faa0-1dfe-43f2-8f3a-199e92ff79e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:37.504334 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.504134 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-workload-certs" pod "2079faa0-1dfe-43f2-8f3a-199e92ff79e7" (UID: "2079faa0-1dfe-43f2-8f3a-199e92ff79e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:37.504334 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.504145 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istiod-ca-cert" pod "2079faa0-1dfe-43f2-8f3a-199e92ff79e7" (UID: "2079faa0-1dfe-43f2-8f3a-199e92ff79e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:02:37.504507 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.504484 4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istiod-ca-cert\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:02:37.504607 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.504514 4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-workload-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:02:37.504607 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.504455 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-credential-socket" pod "2079faa0-1dfe-43f2-8f3a-199e92ff79e7" (UID: "2079faa0-1dfe-43f2-8f3a-199e92ff79e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:37.504607 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.504530 4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-data\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:02:37.504607 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.504546 4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-workload-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:02:37.506165 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.506136 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-podinfo" pod "2079faa0-1dfe-43f2-8f3a-199e92ff79e7" (UID: "2079faa0-1dfe-43f2-8f3a-199e92ff79e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:02:37.506266 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.506192 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-token" pod "2079faa0-1dfe-43f2-8f3a-199e92ff79e7" (UID: "2079faa0-1dfe-43f2-8f3a-199e92ff79e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:02:37.506329 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.506276 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-kube-api-access-cgggv" pod "2079faa0-1dfe-43f2-8f3a-199e92ff79e7" (UID: "2079faa0-1dfe-43f2-8f3a-199e92ff79e7"). InnerVolumeSpecName "kube-api-access-cgggv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:02:37.506407 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.506393 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-envoy" pod "2079faa0-1dfe-43f2-8f3a-199e92ff79e7" (UID: "2079faa0-1dfe-43f2-8f3a-199e92ff79e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:37.605421 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.605393 4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-credential-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:02:37.605421 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.605417 4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-envoy\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:02:37.605421 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.605428 4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-podinfo\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:02:37.605619 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.605436 4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-istio-token\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:02:37.605619 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:37.605445 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgggv\" (UniqueName: \"kubernetes.io/projected/2079faa0-1dfe-43f2-8f3a-199e92ff79e7-kube-api-access-cgggv\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:02:38.009373 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:38.009338 4581 generic.go:356] "Generic (PLEG): container finished" podID="2079faa0-1dfe-43f2-8f3a-199e92ff79e7" containerID="8ce9aa27131079b1f4048ccbf4478c21c5fa6084d56f025ffa6c3291646d2813" exitCode=0 Nov 04 05:02:38.009554 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:38.009390 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" Nov 04 05:02:38.009554 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:38.009419 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" event={"ID":"2079faa0-1dfe-43f2-8f3a-199e92ff79e7","Type":"ContainerDied","Data":"8ce9aa27131079b1f4048ccbf4478c21c5fa6084d56f025ffa6c3291646d2813"} Nov 04 05:02:38.009554 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:38.009474 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7" event={"ID":"2079faa0-1dfe-43f2-8f3a-199e92ff79e7","Type":"ContainerDied","Data":"d2f1988c837ef509ed96323801fecc9ce994ea8d58de11287d5f8008a2ed57a7"} Nov 04 05:02:38.009554 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:38.009496 4581 scope.go:122] "RemoveContainer" containerID="8ce9aa27131079b1f4048ccbf4478c21c5fa6084d56f025ffa6c3291646d2813" Nov 04 05:02:38.020814 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:38.020792 4581 scope.go:122] "RemoveContainer" containerID="8ce9aa27131079b1f4048ccbf4478c21c5fa6084d56f025ffa6c3291646d2813" Nov 04 05:02:38.021146 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:02:38.021127 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ce9aa27131079b1f4048ccbf4478c21c5fa6084d56f025ffa6c3291646d2813\": container with ID starting with 8ce9aa27131079b1f4048ccbf4478c21c5fa6084d56f025ffa6c3291646d2813 not found: ID does not exist" containerID="8ce9aa27131079b1f4048ccbf4478c21c5fa6084d56f025ffa6c3291646d2813" Nov 04 05:02:38.021213 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:38.021155 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ce9aa27131079b1f4048ccbf4478c21c5fa6084d56f025ffa6c3291646d2813"} err="failed to get container status \"8ce9aa27131079b1f4048ccbf4478c21c5fa6084d56f025ffa6c3291646d2813\": rpc error: code = NotFound desc = could not find container \"8ce9aa27131079b1f4048ccbf4478c21c5fa6084d56f025ffa6c3291646d2813\": container with ID starting with 8ce9aa27131079b1f4048ccbf4478c21c5fa6084d56f025ffa6c3291646d2813 not found: ID does not exist" Nov 04 05:02:38.176439 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:38.176404 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7"] Nov 04 05:02:38.225737 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:38.225708 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7"] Nov 04 05:02:39.046552 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:39.046521 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:02:39.230190 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:39.230158 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2079faa0-1dfe-43f2-8f3a-199e92ff79e7" path="/var/lib/kubelet/pods/2079faa0-1dfe-43f2-8f3a-199e92ff79e7/volumes" Nov 04 05:02:43.839828 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:43.839791 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:02:47.716097 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.716063 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd"] Nov 04 05:02:47.716547 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.716535 4581 cpu_manager.go:414] "containerMap: removing container" podUID="2079faa0-1dfe-43f2-8f3a-199e92ff79e7" containerName="istio-proxy" Nov 04 05:02:47.716591 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.716548 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2079faa0-1dfe-43f2-8f3a-199e92ff79e7" containerName="istio-proxy" Nov 04 05:02:47.716661 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.716652 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2079faa0-1dfe-43f2-8f3a-199e92ff79e7" containerName="istio-proxy" Nov 04 05:02:47.721501 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.721479 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:02:47.722620 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.722595 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-model-cache\") pod \"distinct-w4-3eb31c-kserve-89ff97c4b-845sd\" (UID: \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:02:47.722732 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.722639 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-home\") pod \"distinct-w4-3eb31c-kserve-89ff97c4b-845sd\" (UID: \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:02:47.722822 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.722765 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/081b7b2b-8896-4024-bfaf-4dd8afdd893f-tls-certs\") pod \"distinct-w4-3eb31c-kserve-89ff97c4b-845sd\" (UID: \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:02:47.722890 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.722829 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-dshm\") pod \"distinct-w4-3eb31c-kserve-89ff97c4b-845sd\" (UID: \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:02:47.722979 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.722959 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvnpc\" (UniqueName: \"kubernetes.io/projected/081b7b2b-8896-4024-bfaf-4dd8afdd893f-kube-api-access-lvnpc\") pod \"distinct-w4-3eb31c-kserve-89ff97c4b-845sd\" (UID: \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:02:47.723051 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.723030 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-tmp-dir\") pod \"distinct-w4-3eb31c-kserve-89ff97c4b-845sd\" (UID: \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:02:47.777498 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.777468 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd"] Nov 04 05:02:47.801658 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.801625 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:02:47.823788 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.823761 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/081b7b2b-8896-4024-bfaf-4dd8afdd893f-tls-certs\") pod \"distinct-w4-3eb31c-kserve-89ff97c4b-845sd\" (UID: \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:02:47.823926 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.823809 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-dshm\") pod \"distinct-w4-3eb31c-kserve-89ff97c4b-845sd\" (UID: \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:02:47.823926 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.823884 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvnpc\" (UniqueName: \"kubernetes.io/projected/081b7b2b-8896-4024-bfaf-4dd8afdd893f-kube-api-access-lvnpc\") pod \"distinct-w4-3eb31c-kserve-89ff97c4b-845sd\" (UID: \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:02:47.823926 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.823907 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-tmp-dir\") pod \"distinct-w4-3eb31c-kserve-89ff97c4b-845sd\" (UID: \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:02:47.824085 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.823942 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-model-cache\") pod \"distinct-w4-3eb31c-kserve-89ff97c4b-845sd\" (UID: \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:02:47.824085 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.824006 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-home\") pod \"distinct-w4-3eb31c-kserve-89ff97c4b-845sd\" (UID: \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:02:47.824378 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.824339 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3eb31c\"/\"default-dockercfg-pb6qn\"" Nov 04 05:02:47.824485 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.824386 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-tmp-dir\") pod \"distinct-w4-3eb31c-kserve-89ff97c4b-845sd\" (UID: \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:02:47.824485 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.824399 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-model-cache\") pod \"distinct-w4-3eb31c-kserve-89ff97c4b-845sd\" (UID: \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:02:47.824485 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.824416 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3eb31c\"/\"kube-root-ca.crt\"" Nov 04 05:02:47.824485 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.824455 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3eb31c\"/\"openshift-service-ca.crt\"" Nov 04 05:02:47.824485 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.824461 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-home\") pod \"distinct-w4-3eb31c-kserve-89ff97c4b-845sd\" (UID: \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:02:47.824485 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.824459 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3eb31c\"/\"distinct-w4-3eb31c-kserve-self-signed-certs\"" Nov 04 05:02:47.826512 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.826493 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-dshm\") pod \"distinct-w4-3eb31c-kserve-89ff97c4b-845sd\" (UID: \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:02:47.836495 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.836474 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/081b7b2b-8896-4024-bfaf-4dd8afdd893f-tls-certs\") pod \"distinct-w4-3eb31c-kserve-89ff97c4b-845sd\" (UID: \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:02:47.881876 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:47.881787 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvnpc\" (UniqueName: \"kubernetes.io/projected/081b7b2b-8896-4024-bfaf-4dd8afdd893f-kube-api-access-lvnpc\") pod \"distinct-w4-3eb31c-kserve-89ff97c4b-845sd\" (UID: \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:02:47.895193 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:02:47.895152 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 05:02:48.033675 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:48.033580 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:02:48.430467 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:48.430442 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd"] Nov 04 05:02:48.431511 ip-10-0-141-132 kubenswrapper[4581]: W1104 05:02:48.431483 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod081b7b2b_8896_4024_bfaf_4dd8afdd893f.slice/crio-8ea70349cd2e97a06ea17e4d4ea6104d1d4717660282cf8480819f236113b555 WatchSource:0}: Error finding container 8ea70349cd2e97a06ea17e4d4ea6104d1d4717660282cf8480819f236113b555: Status 404 returned error can't find the container with id 8ea70349cd2e97a06ea17e4d4ea6104d1d4717660282cf8480819f236113b555 Nov 04 05:02:48.612693 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:48.612655 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:02:49.073826 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:49.073794 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" event={"ID":"081b7b2b-8896-4024-bfaf-4dd8afdd893f","Type":"ContainerStarted","Data":"5a44e57c33b97b60452e122d0b42bea592ddc54a7b29096b945307f594054b79"} Nov 04 05:02:49.073826 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:49.073837 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" event={"ID":"081b7b2b-8896-4024-bfaf-4dd8afdd893f","Type":"ContainerStarted","Data":"8ea70349cd2e97a06ea17e4d4ea6104d1d4717660282cf8480819f236113b555"} Nov 04 05:02:49.191272 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:49.191218 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" podStartSLOduration=2.191204297 podStartE2EDuration="2.191204297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:02:49.188053273 +0000 UTC m=+928.078910294" watchObservedRunningTime="2026-11-04 05:02:49.191204297 +0000 UTC m=+928.082061318" Nov 04 05:02:50.034083 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:50.034051 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:02:51.558507 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.558476 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2"] Nov 04 05:02:51.562545 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.562523 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.623126 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.623098 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-dab70d-openshift-default-dockercfg-852vc\"" Nov 04 05:02:51.666470 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.666437 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-podinfo\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.666615 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.666510 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4xff\" (UniqueName: \"kubernetes.io/projected/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-kube-api-access-w4xff\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.666672 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.666601 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-workload-certs\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.666672 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.666639 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istiod-ca-cert\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.666766 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.666692 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-workload-socket\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.666817 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.666759 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-envoy\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.666920 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.666899 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-token\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.667061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.667041 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-credential-socket\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.667123 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.667093 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-data\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.674525 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.674485 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2"] Nov 04 05:02:51.767842 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.767813 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-credential-socket\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.768084 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.767863 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-data\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.768084 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.767943 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-podinfo\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.768084 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.768039 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4xff\" (UniqueName: \"kubernetes.io/projected/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-kube-api-access-w4xff\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.768084 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.768071 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-workload-certs\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.768317 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.768097 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istiod-ca-cert\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.768317 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.768152 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-workload-socket\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.768317 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.768181 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-envoy\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.768317 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.768290 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-credential-socket\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.768317 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.768310 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-data\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.768575 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.768361 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-token\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.768575 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.768431 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-workload-certs\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.768575 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.768567 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-workload-socket\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.768870 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.768850 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istiod-ca-cert\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.771028 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.771004 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-envoy\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.771191 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.771174 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-podinfo\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.825745 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.825678 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-token\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.826072 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.826055 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4xff\" (UniqueName: \"kubernetes.io/projected/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-kube-api-access-w4xff\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:51.878221 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:51.878186 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:52.305670 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:52.305638 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2"] Nov 04 05:02:52.307369 ip-10-0-141-132 kubenswrapper[4581]: W1104 05:02:52.307342 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ef48c5_d75d_4a0c_a0d8_43b1d2aec689.slice/crio-32d2a5b6bae458b6e6b568a2c4e3d327ecbc8192e185b62b7f6da3720aab2a9a WatchSource:0}: Error finding container 32d2a5b6bae458b6e6b568a2c4e3d327ecbc8192e185b62b7f6da3720aab2a9a: Status 404 returned error can't find the container with id 32d2a5b6bae458b6e6b568a2c4e3d327ecbc8192e185b62b7f6da3720aab2a9a Nov 04 05:02:52.309745 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:52.309711 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:02:52.309848 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:52.309811 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:02:52.309909 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:52.309868 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:02:53.098818 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:53.098785 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" event={"ID":"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689","Type":"ContainerStarted","Data":"b14948f09af000f39fb24911053dd5000dbf031fc5a7edc74aec637bb47598df"} Nov 04 05:02:53.099261 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:53.098845 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" event={"ID":"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689","Type":"ContainerStarted","Data":"32d2a5b6bae458b6e6b568a2c4e3d327ecbc8192e185b62b7f6da3720aab2a9a"} Nov 04 05:02:53.277017 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:53.267248 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" podStartSLOduration=2.267230008 podStartE2EDuration="2.267230008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:02:53.26666374 +0000 UTC m=+932.157520762" watchObservedRunningTime="2026-11-04 05:02:53.267230008 +0000 UTC m=+932.158087029" Nov 04 05:02:53.337267 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:53.337230 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:02:53.879374 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:53.879342 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:53.881128 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:53.881102 4581 patch_prober.go:29] interesting pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 04 05:02:53.881213 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:53.881166 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" podUID="71ef48c5-d75d-4a0c-a0d8-43b1d2aec689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 04 05:02:54.697517 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:54.697470 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:02:54.878874 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:54.878834 4581 patch_prober.go:29] interesting pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 04 05:02:54.879142 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:54.878894 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" podUID="71ef48c5-d75d-4a0c-a0d8-43b1d2aec689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 04 05:02:55.879522 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:55.879482 4581 patch_prober.go:29] interesting pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 04 05:02:55.879921 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:55.879552 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" podUID="71ef48c5-d75d-4a0c-a0d8-43b1d2aec689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 04 05:02:56.879497 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:56.879460 4581 patch_prober.go:29] interesting pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 04 05:02:56.879675 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:56.879531 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" podUID="71ef48c5-d75d-4a0c-a0d8-43b1d2aec689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 04 05:02:57.896012 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:57.895363 4581 patch_prober.go:29] interesting pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 04 05:02:57.896012 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:57.895427 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" podUID="71ef48c5-d75d-4a0c-a0d8-43b1d2aec689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 04 05:02:58.035116 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:58.035061 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:02:58.047047 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:58.047018 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:02:58.144371 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:58.144340 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:02:58.212430 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:58.212333 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:02:59.173431 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:59.173399 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:59.174249 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:59.174210 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:02:59.175021 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:02:59.174976 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:03:01.431146 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:01.431109 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:02.922436 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:02.922355 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:07.647808 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:07.647767 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:09.404111 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:09.404073 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:12.425824 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:12.425793 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:17.128047 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:17.128008 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:17.887150 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:03:17.887115 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 05:03:21.939880 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:21.939842 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:25.526529 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:25.526495 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:26.929340 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:26.929305 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:31.648885 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:31.648845 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:36.333047 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:36.332979 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:38.616825 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:38.616785 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:41.175962 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:41.175923 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:42.583630 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:42.583597 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:42.719441 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:42.719405 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:43.424646 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:43.424615 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:43.583158 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:43.583122 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m"] Nov 04 05:03:43.678359 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:43.678269 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:44.187792 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:44.187743 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:46.132527 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:46.132491 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:47.886848 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:03:47.886815 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 05:03:49.390402 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:49.390369 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:49.569109 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:49.569081 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:50.278948 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:50.278920 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:50.907647 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:50.907608 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:52.800963 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:52.800934 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:52.928287 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:52.928249 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:54.511235 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:54.511198 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2"] Nov 04 05:03:54.606280 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:54.606249 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:55.907711 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:55.907664 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:56.027297 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:56.027260 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:58.604787 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:58.604743 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" podUID="ec7b6b61-568d-410e-80fb-c51216be67e3" containerName="main" containerID="cri-o://75afb13314116ceeabf8b62fedc915eeb169d8aa8084092cd0a0f836d1d8419e" gracePeriod=45 Nov 04 05:03:58.813718 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:58.813679 4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" podUID="ec7b6b61-568d-410e-80fb-c51216be67e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": dial tcp 10.134.0.56:8000: connect: connection refused" Nov 04 05:03:58.906558 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:58.906534 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:03:58.940008 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:58.939972 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-tmp-dir\") pod \"ec7b6b61-568d-410e-80fb-c51216be67e3\" (UID: \"ec7b6b61-568d-410e-80fb-c51216be67e3\") " Nov 04 05:03:58.940158 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:58.940102 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-dshm\") pod \"ec7b6b61-568d-410e-80fb-c51216be67e3\" (UID: \"ec7b6b61-568d-410e-80fb-c51216be67e3\") " Nov 04 05:03:58.940158 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:58.940133 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec7b6b61-568d-410e-80fb-c51216be67e3-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec7b6b61-568d-410e-80fb-c51216be67e3-tls-certs\") pod \"ec7b6b61-568d-410e-80fb-c51216be67e3\" (UID: \"ec7b6b61-568d-410e-80fb-c51216be67e3\") " Nov 04 05:03:58.940246 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:58.940174 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-home\" (UniqueName: \"kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-home\") pod \"ec7b6b61-568d-410e-80fb-c51216be67e3\" (UID: \"ec7b6b61-568d-410e-80fb-c51216be67e3\") " Nov 04 05:03:58.940246 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:58.940179 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-tmp-dir" pod "ec7b6b61-568d-410e-80fb-c51216be67e3" (UID: "ec7b6b61-568d-410e-80fb-c51216be67e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:58.940306 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:58.940277 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec7b6b61-568d-410e-80fb-c51216be67e3-kube-api-access-kgc4h\" (UniqueName: \"kubernetes.io/projected/ec7b6b61-568d-410e-80fb-c51216be67e3-kube-api-access-kgc4h\") pod \"ec7b6b61-568d-410e-80fb-c51216be67e3\" (UID: \"ec7b6b61-568d-410e-80fb-c51216be67e3\") " Nov 04 05:03:58.940353 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:58.940337 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-model-cache\") pod \"ec7b6b61-568d-410e-80fb-c51216be67e3\" (UID: \"ec7b6b61-568d-410e-80fb-c51216be67e3\") " Nov 04 05:03:58.940480 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:58.940459 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-home" pod "ec7b6b61-568d-410e-80fb-c51216be67e3" (UID: "ec7b6b61-568d-410e-80fb-c51216be67e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:58.940645 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:58.940618 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-model-cache" pod "ec7b6b61-568d-410e-80fb-c51216be67e3" (UID: "ec7b6b61-568d-410e-80fb-c51216be67e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:58.940834 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:58.940818 4581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-model-cache\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:03:58.940895 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:58.940840 4581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-tmp-dir\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:03:58.940895 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:58.940855 4581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-home\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:03:58.942492 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:58.942469 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec7b6b61-568d-410e-80fb-c51216be67e3-tls-certs" pod "ec7b6b61-568d-410e-80fb-c51216be67e3" (UID: "ec7b6b61-568d-410e-80fb-c51216be67e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:03:58.942580 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:58.942554 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-dshm" pod "ec7b6b61-568d-410e-80fb-c51216be67e3" (UID: "ec7b6b61-568d-410e-80fb-c51216be67e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:58.942936 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:58.942921 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec7b6b61-568d-410e-80fb-c51216be67e3-kube-api-access-kgc4h" pod "ec7b6b61-568d-410e-80fb-c51216be67e3" (UID: "ec7b6b61-568d-410e-80fb-c51216be67e3"). InnerVolumeSpecName "kube-api-access-kgc4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:03:59.042393 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.042355 4581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec7b6b61-568d-410e-80fb-c51216be67e3-dshm\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:03:59.042393 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.042383 4581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec7b6b61-568d-410e-80fb-c51216be67e3-tls-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:03:59.042393 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.042394 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgc4h\" (UniqueName: \"kubernetes.io/projected/ec7b6b61-568d-410e-80fb-c51216be67e3-kube-api-access-kgc4h\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:03:59.469184 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.469156 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7"] Nov 04 05:03:59.469729 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.469716 4581 cpu_manager.go:414] "containerMap: removing container" podUID="ec7b6b61-568d-410e-80fb-c51216be67e3" containerName="main" Nov 04 05:03:59.469778 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.469731 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec7b6b61-568d-410e-80fb-c51216be67e3" containerName="main" Nov 04 05:03:59.469850 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.469833 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec7b6b61-568d-410e-80fb-c51216be67e3" containerName="main" Nov 04 05:03:59.473244 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.473229 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:03:59.491134 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.491073 4581 generic.go:356] "Generic (PLEG): container finished" podID="ec7b6b61-568d-410e-80fb-c51216be67e3" containerID="75afb13314116ceeabf8b62fedc915eeb169d8aa8084092cd0a0f836d1d8419e" exitCode=0 Nov 04 05:03:59.491134 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.491118 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" Nov 04 05:03:59.491272 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.491147 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" event={"ID":"ec7b6b61-568d-410e-80fb-c51216be67e3","Type":"ContainerDied","Data":"75afb13314116ceeabf8b62fedc915eeb169d8aa8084092cd0a0f836d1d8419e"} Nov 04 05:03:59.491272 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.491188 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m" event={"ID":"ec7b6b61-568d-410e-80fb-c51216be67e3","Type":"ContainerDied","Data":"424f76a097e3cda3b74714744f51726460d4f0edafbb552d8d97123d342d371d"} Nov 04 05:03:59.491272 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.491206 4581 scope.go:122] "RemoveContainer" containerID="75afb13314116ceeabf8b62fedc915eeb169d8aa8084092cd0a0f836d1d8419e" Nov 04 05:03:59.501757 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.501732 4581 scope.go:122] "RemoveContainer" containerID="75afb13314116ceeabf8b62fedc915eeb169d8aa8084092cd0a0f836d1d8419e" Nov 04 05:03:59.502160 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:03:59.502141 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75afb13314116ceeabf8b62fedc915eeb169d8aa8084092cd0a0f836d1d8419e\": container with ID starting with 75afb13314116ceeabf8b62fedc915eeb169d8aa8084092cd0a0f836d1d8419e not found: ID does not exist" containerID="75afb13314116ceeabf8b62fedc915eeb169d8aa8084092cd0a0f836d1d8419e" Nov 04 05:03:59.502274 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.502169 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75afb13314116ceeabf8b62fedc915eeb169d8aa8084092cd0a0f836d1d8419e"} err="failed to get container status \"75afb13314116ceeabf8b62fedc915eeb169d8aa8084092cd0a0f836d1d8419e\": rpc error: code = NotFound desc = could not find container \"75afb13314116ceeabf8b62fedc915eeb169d8aa8084092cd0a0f836d1d8419e\": container with ID starting with 75afb13314116ceeabf8b62fedc915eeb169d8aa8084092cd0a0f836d1d8419e not found: ID does not exist" Nov 04 05:03:59.532139 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.532109 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7"] Nov 04 05:03:59.547529 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.547494 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de6539ee-da6d-40c5-917b-1ecdee1b133c-tls-certs\") pod \"distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7\" (UID: \"de6539ee-da6d-40c5-917b-1ecdee1b133c\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:03:59.547529 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.547530 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-dshm\") pod \"distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7\" (UID: \"de6539ee-da6d-40c5-917b-1ecdee1b133c\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:03:59.547876 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.547588 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-home\") pod \"distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7\" (UID: \"de6539ee-da6d-40c5-917b-1ecdee1b133c\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:03:59.547876 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.547698 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg9pt\" (UniqueName: \"kubernetes.io/projected/de6539ee-da6d-40c5-917b-1ecdee1b133c-kube-api-access-kg9pt\") pod \"distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7\" (UID: \"de6539ee-da6d-40c5-917b-1ecdee1b133c\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:03:59.547876 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.547736 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-model-cache\") pod \"distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7\" (UID: \"de6539ee-da6d-40c5-917b-1ecdee1b133c\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:03:59.547876 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.547864 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-tmp-dir\") pod \"distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7\" (UID: \"de6539ee-da6d-40c5-917b-1ecdee1b133c\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:03:59.548950 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.548924 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:03:59.578481 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.578456 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3eb31c\"/\"distinct-2-w4-3eb31c-kserve-self-signed-certs\"" Nov 04 05:03:59.649424 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.649395 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-tmp-dir\") pod \"distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7\" (UID: \"de6539ee-da6d-40c5-917b-1ecdee1b133c\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:03:59.649904 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.649481 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de6539ee-da6d-40c5-917b-1ecdee1b133c-tls-certs\") pod \"distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7\" (UID: \"de6539ee-da6d-40c5-917b-1ecdee1b133c\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:03:59.649904 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.649502 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-dshm\") pod \"distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7\" (UID: \"de6539ee-da6d-40c5-917b-1ecdee1b133c\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:03:59.649904 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.649543 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-home\") pod \"distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7\" (UID: \"de6539ee-da6d-40c5-917b-1ecdee1b133c\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:03:59.649904 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.649563 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg9pt\" (UniqueName: \"kubernetes.io/projected/de6539ee-da6d-40c5-917b-1ecdee1b133c-kube-api-access-kg9pt\") pod \"distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7\" (UID: \"de6539ee-da6d-40c5-917b-1ecdee1b133c\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:03:59.649904 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.649695 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-model-cache\") pod \"distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7\" (UID: \"de6539ee-da6d-40c5-917b-1ecdee1b133c\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:03:59.649904 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.649787 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-tmp-dir\") pod \"distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7\" (UID: \"de6539ee-da6d-40c5-917b-1ecdee1b133c\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:03:59.650187 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.649925 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-home\") pod \"distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7\" (UID: \"de6539ee-da6d-40c5-917b-1ecdee1b133c\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:03:59.650187 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.650029 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-model-cache\") pod \"distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7\" (UID: \"de6539ee-da6d-40c5-917b-1ecdee1b133c\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:03:59.652067 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.652040 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-dshm\") pod \"distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7\" (UID: \"de6539ee-da6d-40c5-917b-1ecdee1b133c\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:03:59.652349 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.652330 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de6539ee-da6d-40c5-917b-1ecdee1b133c-tls-certs\") pod \"distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7\" (UID: \"de6539ee-da6d-40c5-917b-1ecdee1b133c\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:03:59.704237 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.704207 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m"] Nov 04 05:03:59.710227 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.710200 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0aae2f/premium-w0-0aae2f-kserve-5bcd79c557-xss8m"] Nov 04 05:03:59.710392 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.710374 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg9pt\" (UniqueName: \"kubernetes.io/projected/de6539ee-da6d-40c5-917b-1ecdee1b133c-kube-api-access-kg9pt\") pod \"distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7\" (UID: \"de6539ee-da6d-40c5-917b-1ecdee1b133c\") " pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:03:59.785458 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:03:59.785384 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:04:00.179317 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:00.179285 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7"] Nov 04 05:04:00.180422 ip-10-0-141-132 kubenswrapper[4581]: W1104 05:04:00.180396 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde6539ee_da6d_40c5_917b_1ecdee1b133c.slice/crio-78e8dda970acc59f81656e5c529ecadf046eb5ae7529c1d6ecf187cb654a4b15 WatchSource:0}: Error finding container 78e8dda970acc59f81656e5c529ecadf046eb5ae7529c1d6ecf187cb654a4b15: Status 404 returned error can't find the container with id 78e8dda970acc59f81656e5c529ecadf046eb5ae7529c1d6ecf187cb654a4b15 Nov 04 05:04:00.499835 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:00.499743 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" event={"ID":"de6539ee-da6d-40c5-917b-1ecdee1b133c","Type":"ContainerStarted","Data":"27532f458c59a2d88879ad2b44a4342c26975abc946bb60285108cf9ca3d1ca2"} Nov 04 05:04:00.499835 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:00.499791 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" event={"ID":"de6539ee-da6d-40c5-917b-1ecdee1b133c","Type":"ContainerStarted","Data":"78e8dda970acc59f81656e5c529ecadf046eb5ae7529c1d6ecf187cb654a4b15"} Nov 04 05:04:00.615713 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:00.615657 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" podStartSLOduration=1.6156420790000001 podStartE2EDuration="1.615642079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:04:00.615190737 +0000 UTC m=+999.506047758" watchObservedRunningTime="2026-11-04 05:04:00.615642079 +0000 UTC m=+999.506499105" Nov 04 05:04:00.722142 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:00.722099 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:00.787131 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:00.787033 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:04:01.172833 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:01.172800 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:01.231724 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:01.231691 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec7b6b61-568d-410e-80fb-c51216be67e3" path="/var/lib/kubelet/pods/ec7b6b61-568d-410e-80fb-c51216be67e3/volumes" Nov 04 05:04:01.486819 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:01.486723 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:01.806694 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:01.806591 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:05.536705 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:05.536654 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:06.400097 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:06.400061 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7"] Nov 04 05:04:06.933755 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:06.933725 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:07.061464 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:07.061424 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:07.642297 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:07.642263 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:09.537092 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:09.537035 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" podUID="1c7c486a-863f-49ff-abb3-819e3905e640" containerName="main" containerID="cri-o://2c450522cd19ab5f613fc4d66a9d9cfe290dc9f833a892624e1dc735e70571fc" gracePeriod=45 Nov 04 05:04:09.785489 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:09.785463 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:04:09.837860 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:09.837811 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:04:09.863344 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:09.863256 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:04:09.976707 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:09.976677 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c7c486a-863f-49ff-abb3-819e3905e640-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c7c486a-863f-49ff-abb3-819e3905e640-tls-certs\") pod \"1c7c486a-863f-49ff-abb3-819e3905e640\" (UID: \"1c7c486a-863f-49ff-abb3-819e3905e640\") " Nov 04 05:04:09.976918 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:09.976732 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-dshm\") pod \"1c7c486a-863f-49ff-abb3-819e3905e640\" (UID: \"1c7c486a-863f-49ff-abb3-819e3905e640\") " Nov 04 05:04:09.976918 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:09.976783 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-home\" (UniqueName: \"kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-home\") pod \"1c7c486a-863f-49ff-abb3-819e3905e640\" (UID: \"1c7c486a-863f-49ff-abb3-819e3905e640\") " Nov 04 05:04:09.976918 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:09.976831 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-tmp-dir\") pod \"1c7c486a-863f-49ff-abb3-819e3905e640\" (UID: \"1c7c486a-863f-49ff-abb3-819e3905e640\") " Nov 04 05:04:09.976918 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:09.976896 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-model-cache\") pod \"1c7c486a-863f-49ff-abb3-819e3905e640\" (UID: \"1c7c486a-863f-49ff-abb3-819e3905e640\") " Nov 04 05:04:09.977196 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:09.977050 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-home" pod "1c7c486a-863f-49ff-abb3-819e3905e640" (UID: "1c7c486a-863f-49ff-abb3-819e3905e640"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:09.977196 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:09.977068 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c7c486a-863f-49ff-abb3-819e3905e640-kube-api-access-dqdfv\" (UniqueName: \"kubernetes.io/projected/1c7c486a-863f-49ff-abb3-819e3905e640-kube-api-access-dqdfv\") pod \"1c7c486a-863f-49ff-abb3-819e3905e640\" (UID: \"1c7c486a-863f-49ff-abb3-819e3905e640\") " Nov 04 05:04:09.977293 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:09.977176 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-tmp-dir" pod "1c7c486a-863f-49ff-abb3-819e3905e640" (UID: "1c7c486a-863f-49ff-abb3-819e3905e640"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:09.977293 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:09.977227 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-model-cache" pod "1c7c486a-863f-49ff-abb3-819e3905e640" (UID: "1c7c486a-863f-49ff-abb3-819e3905e640"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:09.977921 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:09.977828 4581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-tmp-dir\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:09.977921 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:09.977866 4581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-model-cache\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:09.977921 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:09.977896 4581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-home\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:09.979375 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:09.979347 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c7c486a-863f-49ff-abb3-819e3905e640-kube-api-access-dqdfv" pod "1c7c486a-863f-49ff-abb3-819e3905e640" (UID: "1c7c486a-863f-49ff-abb3-819e3905e640"). InnerVolumeSpecName "kube-api-access-dqdfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:04:09.979375 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:09.979362 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-dshm" pod "1c7c486a-863f-49ff-abb3-819e3905e640" (UID: "1c7c486a-863f-49ff-abb3-819e3905e640"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:09.979375 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:09.979365 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c7c486a-863f-49ff-abb3-819e3905e640-tls-certs" pod "1c7c486a-863f-49ff-abb3-819e3905e640" (UID: "1c7c486a-863f-49ff-abb3-819e3905e640"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:04:10.053194 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:10.053085 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:10.079472 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:10.079444 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqdfv\" (UniqueName: \"kubernetes.io/projected/1c7c486a-863f-49ff-abb3-819e3905e640-kube-api-access-dqdfv\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:10.079472 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:10.079470 4581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c7c486a-863f-49ff-abb3-819e3905e640-tls-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:10.079668 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:10.079480 4581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c7c486a-863f-49ff-abb3-819e3905e640-dshm\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:10.564755 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:10.564724 4581 generic.go:356] "Generic (PLEG): container finished" podID="1c7c486a-863f-49ff-abb3-819e3905e640" containerID="2c450522cd19ab5f613fc4d66a9d9cfe290dc9f833a892624e1dc735e70571fc" exitCode=0 Nov 04 05:04:10.565289 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:10.564817 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" Nov 04 05:04:10.565289 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:10.564826 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" event={"ID":"1c7c486a-863f-49ff-abb3-819e3905e640","Type":"ContainerDied","Data":"2c450522cd19ab5f613fc4d66a9d9cfe290dc9f833a892624e1dc735e70571fc"} Nov 04 05:04:10.565289 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:10.564880 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" event={"ID":"1c7c486a-863f-49ff-abb3-819e3905e640","Type":"ContainerDied","Data":"0ce6616246c72a56d74fa53b93090b1f089127aa8df0bf16c8fec7f82c6f0dc1"} Nov 04 05:04:10.565289 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:10.564910 4581 scope.go:122] "RemoveContainer" containerID="2c450522cd19ab5f613fc4d66a9d9cfe290dc9f833a892624e1dc735e70571fc" Nov 04 05:04:10.578596 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:10.578577 4581 scope.go:122] "RemoveContainer" containerID="2c450522cd19ab5f613fc4d66a9d9cfe290dc9f833a892624e1dc735e70571fc" Nov 04 05:04:10.578931 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:04:10.578907 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c450522cd19ab5f613fc4d66a9d9cfe290dc9f833a892624e1dc735e70571fc\": container with ID starting with 2c450522cd19ab5f613fc4d66a9d9cfe290dc9f833a892624e1dc735e70571fc not found: ID does not exist" containerID="2c450522cd19ab5f613fc4d66a9d9cfe290dc9f833a892624e1dc735e70571fc" Nov 04 05:04:10.579011 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:10.578943 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c450522cd19ab5f613fc4d66a9d9cfe290dc9f833a892624e1dc735e70571fc"} err="failed to get container status \"2c450522cd19ab5f613fc4d66a9d9cfe290dc9f833a892624e1dc735e70571fc\": rpc error: code = NotFound desc = could not find container \"2c450522cd19ab5f613fc4d66a9d9cfe290dc9f833a892624e1dc735e70571fc\": container with ID starting with 2c450522cd19ab5f613fc4d66a9d9cfe290dc9f833a892624e1dc735e70571fc not found: ID does not exist" Nov 04 05:04:10.581303 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:10.581284 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:04:10.727535 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:10.727492 4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2" podUID="1c7c486a-863f-49ff-abb3-819e3905e640" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 04 05:04:10.731009 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:10.730969 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2"] Nov 04 05:04:10.780921 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:10.780887 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-1874d8/premium-w1-1874d8-kserve-586794cbc9-vvht2"] Nov 04 05:04:11.230285 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:11.230257 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c7c486a-863f-49ff-abb3-819e3905e640" path="/var/lib/kubelet/pods/1c7c486a-863f-49ff-abb3-819e3905e640/volumes" Nov 04 05:04:11.401141 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:11.401097 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7" podUID="4c58df41-54f0-4939-ae70-0e9edb402458" containerName="payload-processing" containerID="cri-o://c2c4f3660d7224ffe5a20eb59d954baaa05727faf40d99caffc5c7cd760038f0" gracePeriod=25 Nov 04 05:04:11.940780 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:11.940750 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:13.347177 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:13.347138 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:16.453980 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:16.453944 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:16.607670 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:16.607635 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:16.747408 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:16.747314 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:17.243795 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:17.243755 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:17.887180 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:04:17.887148 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 05:04:18.004131 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:18.004098 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6"] Nov 04 05:04:18.004371 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:18.004334 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" podUID="510d29ba-d057-4f6b-af44-aa85ab0c4ed6" containerName="istio-proxy" containerID="cri-o://9ba1f9573a3f0a49bdb5ae38cdd71b729b4ab628b844fe9e3729eaa60e04b0a0" gracePeriod=30 Nov 04 05:04:19.486075 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:19.486033 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:20.557065 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:20.556974 4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7" podUID="4c58df41-54f0-4939-ae70-0e9edb402458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 05:04:20.951251 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:20.951219 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:21.131917 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:21.131884 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:21.544981 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:21.544946 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:21.739066 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:21.738942 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:22.479386 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:22.479345 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:23.284700 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.284674 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 05:04:23.343624 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.343590 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-workload-certs\") pod \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " Nov 04 05:04:23.343775 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.343635 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-data\") pod \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " Nov 04 05:04:23.343775 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.343672 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-envoy\") pod \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " Nov 04 05:04:23.343775 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.343725 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-podinfo\") pod \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " Nov 04 05:04:23.343775 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.343750 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-token\" (UniqueName: \"kubernetes.io/projected/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-token\") pod \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " Nov 04 05:04:23.343971 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.343803 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-credential-socket\") pod \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " Nov 04 05:04:23.343971 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.343855 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-workload-certs" pod "510d29ba-d057-4f6b-af44-aa85ab0c4ed6" (UID: "510d29ba-d057-4f6b-af44-aa85ab0c4ed6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:23.343971 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.343926 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istiod-ca-cert\") pod \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " Nov 04 05:04:23.343971 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.343961 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-data" pod "510d29ba-d057-4f6b-af44-aa85ab0c4ed6" (UID: "510d29ba-d057-4f6b-af44-aa85ab0c4ed6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:23.344217 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.343967 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-kube-api-access-49r9w\" (UniqueName: \"kubernetes.io/projected/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-kube-api-access-49r9w\") pod \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " Nov 04 05:04:23.344217 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.344119 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-credential-socket" pod "510d29ba-d057-4f6b-af44-aa85ab0c4ed6" (UID: "510d29ba-d057-4f6b-af44-aa85ab0c4ed6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:23.344217 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.344151 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-workload-socket\") pod \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\" (UID: \"510d29ba-d057-4f6b-af44-aa85ab0c4ed6\") " Nov 04 05:04:23.344389 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.344286 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istiod-ca-cert" pod "510d29ba-d057-4f6b-af44-aa85ab0c4ed6" (UID: "510d29ba-d057-4f6b-af44-aa85ab0c4ed6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:04:23.344452 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.344381 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-workload-socket" pod "510d29ba-d057-4f6b-af44-aa85ab0c4ed6" (UID: "510d29ba-d057-4f6b-af44-aa85ab0c4ed6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:23.344879 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.344855 4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-credential-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:23.344973 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.344885 4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istiod-ca-cert\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:23.344973 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.344900 4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-workload-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:23.344973 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.344915 4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-workload-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:23.344973 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.344930 4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-data\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:23.346187 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.346158 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-podinfo" pod "510d29ba-d057-4f6b-af44-aa85ab0c4ed6" (UID: "510d29ba-d057-4f6b-af44-aa85ab0c4ed6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:04:23.346187 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.346163 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-token" pod "510d29ba-d057-4f6b-af44-aa85ab0c4ed6" (UID: "510d29ba-d057-4f6b-af44-aa85ab0c4ed6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:04:23.346345 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.346205 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-envoy" pod "510d29ba-d057-4f6b-af44-aa85ab0c4ed6" (UID: "510d29ba-d057-4f6b-af44-aa85ab0c4ed6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:23.346345 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.346231 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-kube-api-access-49r9w" pod "510d29ba-d057-4f6b-af44-aa85ab0c4ed6" (UID: "510d29ba-d057-4f6b-af44-aa85ab0c4ed6"). InnerVolumeSpecName "kube-api-access-49r9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:04:23.445837 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.445806 4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-envoy\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:23.445837 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.445832 4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-podinfo\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:23.446039 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.445849 4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-istio-token\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:23.446039 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.445859 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49r9w\" (UniqueName: \"kubernetes.io/projected/510d29ba-d057-4f6b-af44-aa85ab0c4ed6-kube-api-access-49r9w\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:23.643716 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.643680 4581 generic.go:356] "Generic (PLEG): container finished" podID="510d29ba-d057-4f6b-af44-aa85ab0c4ed6" containerID="9ba1f9573a3f0a49bdb5ae38cdd71b729b4ab628b844fe9e3729eaa60e04b0a0" exitCode=0 Nov 04 05:04:23.643926 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.643735 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" Nov 04 05:04:23.643926 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.643766 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" event={"ID":"510d29ba-d057-4f6b-af44-aa85ab0c4ed6","Type":"ContainerDied","Data":"9ba1f9573a3f0a49bdb5ae38cdd71b729b4ab628b844fe9e3729eaa60e04b0a0"} Nov 04 05:04:23.643926 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.643826 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6" event={"ID":"510d29ba-d057-4f6b-af44-aa85ab0c4ed6","Type":"ContainerDied","Data":"3cfd8451a206acb80e7a0c59028d4626a37e2242c898fb0cf3f14d4cc0d1755e"} Nov 04 05:04:23.643926 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.643855 4581 scope.go:122] "RemoveContainer" containerID="9ba1f9573a3f0a49bdb5ae38cdd71b729b4ab628b844fe9e3729eaa60e04b0a0" Nov 04 05:04:23.654831 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.654816 4581 scope.go:122] "RemoveContainer" containerID="9ba1f9573a3f0a49bdb5ae38cdd71b729b4ab628b844fe9e3729eaa60e04b0a0" Nov 04 05:04:23.655149 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:04:23.655130 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba1f9573a3f0a49bdb5ae38cdd71b729b4ab628b844fe9e3729eaa60e04b0a0\": container with ID starting with 9ba1f9573a3f0a49bdb5ae38cdd71b729b4ab628b844fe9e3729eaa60e04b0a0 not found: ID does not exist" containerID="9ba1f9573a3f0a49bdb5ae38cdd71b729b4ab628b844fe9e3729eaa60e04b0a0" Nov 04 05:04:23.655227 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.655159 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba1f9573a3f0a49bdb5ae38cdd71b729b4ab628b844fe9e3729eaa60e04b0a0"} err="failed to get container status \"9ba1f9573a3f0a49bdb5ae38cdd71b729b4ab628b844fe9e3729eaa60e04b0a0\": rpc error: code = NotFound desc = could not find container \"9ba1f9573a3f0a49bdb5ae38cdd71b729b4ab628b844fe9e3729eaa60e04b0a0\": container with ID starting with 9ba1f9573a3f0a49bdb5ae38cdd71b729b4ab628b844fe9e3729eaa60e04b0a0 not found: ID does not exist" Nov 04 05:04:23.810978 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.810948 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6"] Nov 04 05:04:23.815142 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:23.815118 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6"] Nov 04 05:04:24.632484 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:24.632446 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:25.138789 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:25.138747 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:25.158810 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:25.158776 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285"] Nov 04 05:04:25.159200 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:25.159168 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" podUID="aa8dcd7f-0f9f-4eef-a831-8d164c92830e" containerName="istio-proxy" containerID="cri-o://d8df4337b2e39c01778326cb4b841029a4b68f6c1d575c9d5056d087911c0abf" gracePeriod=30 Nov 04 05:04:25.237153 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:25.237118 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="510d29ba-d057-4f6b-af44-aa85ab0c4ed6" path="/var/lib/kubelet/pods/510d29ba-d057-4f6b-af44-aa85ab0c4ed6/volumes" Nov 04 05:04:26.402718 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:26.402680 4581 patch_prober.go:29] interesting pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 05:04:26.403139 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:26.402750 4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" podUID="aa8dcd7f-0f9f-4eef-a831-8d164c92830e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 05:04:26.762483 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:26.762386 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:27.636663 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:27.636627 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:27.824254 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:27.824223 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:30.429121 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.429033 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 05:04:30.523143 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.523110 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-credential-socket\") pod \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " Nov 04 05:04:30.523143 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.523144 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-data\") pod \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " Nov 04 05:04:30.523356 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.523195 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-envoy\") pod \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " Nov 04 05:04:30.523356 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.523246 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-workload-socket\") pod \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " Nov 04 05:04:30.523356 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.523330 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-workload-certs\") pod \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " Nov 04 05:04:30.523513 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.523365 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istiod-ca-cert\") pod \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " Nov 04 05:04:30.523513 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.523459 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-kube-api-access-fjxcs\" (UniqueName: \"kubernetes.io/projected/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-kube-api-access-fjxcs\") pod \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " Nov 04 05:04:30.523631 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.523521 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-podinfo\") pod \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " Nov 04 05:04:30.523631 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.523436 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-credential-socket" pod "aa8dcd7f-0f9f-4eef-a831-8d164c92830e" (UID: "aa8dcd7f-0f9f-4eef-a831-8d164c92830e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:30.523631 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.523574 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-token\" (UniqueName: \"kubernetes.io/projected/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-token\") pod \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\" (UID: \"aa8dcd7f-0f9f-4eef-a831-8d164c92830e\") " Nov 04 05:04:30.523631 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.523516 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-data" pod "aa8dcd7f-0f9f-4eef-a831-8d164c92830e" (UID: "aa8dcd7f-0f9f-4eef-a831-8d164c92830e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:30.523824 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.523627 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-workload-certs" pod "aa8dcd7f-0f9f-4eef-a831-8d164c92830e" (UID: "aa8dcd7f-0f9f-4eef-a831-8d164c92830e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:30.523824 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.523648 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-workload-socket" pod "aa8dcd7f-0f9f-4eef-a831-8d164c92830e" (UID: "aa8dcd7f-0f9f-4eef-a831-8d164c92830e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:30.523931 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.523847 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istiod-ca-cert" pod "aa8dcd7f-0f9f-4eef-a831-8d164c92830e" (UID: "aa8dcd7f-0f9f-4eef-a831-8d164c92830e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:04:30.524127 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.524105 4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-workload-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:30.524245 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.524133 4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-workload-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:30.524245 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.524150 4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istiod-ca-cert\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:30.524245 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.524166 4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-credential-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:30.524245 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.524190 4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-data\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:30.525909 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.525878 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-token" pod "aa8dcd7f-0f9f-4eef-a831-8d164c92830e" (UID: "aa8dcd7f-0f9f-4eef-a831-8d164c92830e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:04:30.526065 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.525921 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-podinfo" pod "aa8dcd7f-0f9f-4eef-a831-8d164c92830e" (UID: "aa8dcd7f-0f9f-4eef-a831-8d164c92830e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:04:30.526065 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.525932 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-envoy" pod "aa8dcd7f-0f9f-4eef-a831-8d164c92830e" (UID: "aa8dcd7f-0f9f-4eef-a831-8d164c92830e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:30.526065 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.525945 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-kube-api-access-fjxcs" pod "aa8dcd7f-0f9f-4eef-a831-8d164c92830e" (UID: "aa8dcd7f-0f9f-4eef-a831-8d164c92830e"). InnerVolumeSpecName "kube-api-access-fjxcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:04:30.556246 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.556214 4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7" podUID="4c58df41-54f0-4939-ae70-0e9edb402458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 05:04:30.625505 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.625474 4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-envoy\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:30.625505 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.625501 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjxcs\" (UniqueName: \"kubernetes.io/projected/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-kube-api-access-fjxcs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:30.625505 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.625511 4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-podinfo\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:30.625738 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.625519 4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa8dcd7f-0f9f-4eef-a831-8d164c92830e-istio-token\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:30.683820 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.683735 4581 generic.go:356] "Generic (PLEG): container finished" podID="aa8dcd7f-0f9f-4eef-a831-8d164c92830e" containerID="d8df4337b2e39c01778326cb4b841029a4b68f6c1d575c9d5056d087911c0abf" exitCode=0 Nov 04 05:04:30.683820 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.683798 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" Nov 04 05:04:30.683820 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.683809 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" event={"ID":"aa8dcd7f-0f9f-4eef-a831-8d164c92830e","Type":"ContainerDied","Data":"d8df4337b2e39c01778326cb4b841029a4b68f6c1d575c9d5056d087911c0abf"} Nov 04 05:04:30.684094 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.683900 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285" event={"ID":"aa8dcd7f-0f9f-4eef-a831-8d164c92830e","Type":"ContainerDied","Data":"219fb0413d8880e02ecfff33866226caf2269cd96672debc91394a8d08b8d893"} Nov 04 05:04:30.684094 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.683923 4581 scope.go:122] "RemoveContainer" containerID="d8df4337b2e39c01778326cb4b841029a4b68f6c1d575c9d5056d087911c0abf" Nov 04 05:04:30.694780 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.694759 4581 scope.go:122] "RemoveContainer" containerID="33c3a7763a73a9cc67e554288a78ac02b3ae5e024361baf5d16f26a0894cc30c" Nov 04 05:04:30.704904 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.704886 4581 scope.go:122] "RemoveContainer" containerID="d8df4337b2e39c01778326cb4b841029a4b68f6c1d575c9d5056d087911c0abf" Nov 04 05:04:30.705220 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:04:30.705201 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8df4337b2e39c01778326cb4b841029a4b68f6c1d575c9d5056d087911c0abf\": container with ID starting with d8df4337b2e39c01778326cb4b841029a4b68f6c1d575c9d5056d087911c0abf not found: ID does not exist" containerID="d8df4337b2e39c01778326cb4b841029a4b68f6c1d575c9d5056d087911c0abf" Nov 04 05:04:30.705307 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.705229 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8df4337b2e39c01778326cb4b841029a4b68f6c1d575c9d5056d087911c0abf"} err="failed to get container status \"d8df4337b2e39c01778326cb4b841029a4b68f6c1d575c9d5056d087911c0abf\": rpc error: code = NotFound desc = could not find container \"d8df4337b2e39c01778326cb4b841029a4b68f6c1d575c9d5056d087911c0abf\": container with ID starting with d8df4337b2e39c01778326cb4b841029a4b68f6c1d575c9d5056d087911c0abf not found: ID does not exist" Nov 04 05:04:30.705307 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.705248 4581 scope.go:122] "RemoveContainer" containerID="33c3a7763a73a9cc67e554288a78ac02b3ae5e024361baf5d16f26a0894cc30c" Nov 04 05:04:30.705542 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:04:30.705524 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c3a7763a73a9cc67e554288a78ac02b3ae5e024361baf5d16f26a0894cc30c\": container with ID starting with 33c3a7763a73a9cc67e554288a78ac02b3ae5e024361baf5d16f26a0894cc30c not found: ID does not exist" containerID="33c3a7763a73a9cc67e554288a78ac02b3ae5e024361baf5d16f26a0894cc30c" Nov 04 05:04:30.705584 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.705551 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c3a7763a73a9cc67e554288a78ac02b3ae5e024361baf5d16f26a0894cc30c"} err="failed to get container status \"33c3a7763a73a9cc67e554288a78ac02b3ae5e024361baf5d16f26a0894cc30c\": rpc error: code = NotFound desc = could not find container \"33c3a7763a73a9cc67e554288a78ac02b3ae5e024361baf5d16f26a0894cc30c\": container with ID starting with 33c3a7763a73a9cc67e554288a78ac02b3ae5e024361baf5d16f26a0894cc30c not found: ID does not exist" Nov 04 05:04:30.851731 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.851696 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285"] Nov 04 05:04:30.855972 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:30.855945 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285"] Nov 04 05:04:31.230364 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:31.230332 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa8dcd7f-0f9f-4eef-a831-8d164c92830e" path="/var/lib/kubelet/pods/aa8dcd7f-0f9f-4eef-a831-8d164c92830e/volumes" Nov 04 05:04:31.670967 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:31.670944 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7" Nov 04 05:04:31.691747 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:31.691713 4581 generic.go:356] "Generic (PLEG): container finished" podID="4c58df41-54f0-4939-ae70-0e9edb402458" containerID="c2c4f3660d7224ffe5a20eb59d954baaa05727faf40d99caffc5c7cd760038f0" exitCode=0 Nov 04 05:04:31.691883 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:31.691757 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7" Nov 04 05:04:31.691883 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:31.691796 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7" event={"ID":"4c58df41-54f0-4939-ae70-0e9edb402458","Type":"ContainerDied","Data":"c2c4f3660d7224ffe5a20eb59d954baaa05727faf40d99caffc5c7cd760038f0"} Nov 04 05:04:31.691883 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:31.691843 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7" event={"ID":"4c58df41-54f0-4939-ae70-0e9edb402458","Type":"ContainerDied","Data":"4287d0088315a63955550e2d40458b02bd83e012bfbe610b38a35aeb143c35c0"} Nov 04 05:04:31.691883 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:31.691865 4581 scope.go:122] "RemoveContainer" containerID="c2c4f3660d7224ffe5a20eb59d954baaa05727faf40d99caffc5c7cd760038f0" Nov 04 05:04:31.701435 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:31.701412 4581 scope.go:122] "RemoveContainer" containerID="c2c4f3660d7224ffe5a20eb59d954baaa05727faf40d99caffc5c7cd760038f0" Nov 04 05:04:31.701734 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:04:31.701711 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c4f3660d7224ffe5a20eb59d954baaa05727faf40d99caffc5c7cd760038f0\": container with ID starting with c2c4f3660d7224ffe5a20eb59d954baaa05727faf40d99caffc5c7cd760038f0 not found: ID does not exist" containerID="c2c4f3660d7224ffe5a20eb59d954baaa05727faf40d99caffc5c7cd760038f0" Nov 04 05:04:31.701800 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:31.701745 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c4f3660d7224ffe5a20eb59d954baaa05727faf40d99caffc5c7cd760038f0"} err="failed to get container status \"c2c4f3660d7224ffe5a20eb59d954baaa05727faf40d99caffc5c7cd760038f0\": rpc error: code = NotFound desc = could not find container \"c2c4f3660d7224ffe5a20eb59d954baaa05727faf40d99caffc5c7cd760038f0\": container with ID starting with c2c4f3660d7224ffe5a20eb59d954baaa05727faf40d99caffc5c7cd760038f0 not found: ID does not exist" Nov 04 05:04:31.737501 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:31.737470 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c58df41-54f0-4939-ae70-0e9edb402458-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c58df41-54f0-4939-ae70-0e9edb402458-plugins-config-volume\") pod \"4c58df41-54f0-4939-ae70-0e9edb402458\" (UID: \"4c58df41-54f0-4939-ae70-0e9edb402458\") " Nov 04 05:04:31.737672 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:31.737637 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c58df41-54f0-4939-ae70-0e9edb402458-kube-api-access-x7b97\" (UniqueName: \"kubernetes.io/projected/4c58df41-54f0-4939-ae70-0e9edb402458-kube-api-access-x7b97\") pod \"4c58df41-54f0-4939-ae70-0e9edb402458\" (UID: \"4c58df41-54f0-4939-ae70-0e9edb402458\") " Nov 04 05:04:31.737889 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:31.737867 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c58df41-54f0-4939-ae70-0e9edb402458-plugins-config-volume" pod "4c58df41-54f0-4939-ae70-0e9edb402458" (UID: "4c58df41-54f0-4939-ae70-0e9edb402458"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:04:31.738163 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:31.738147 4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c58df41-54f0-4939-ae70-0e9edb402458-plugins-config-volume\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:31.744715 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:31.744690 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c58df41-54f0-4939-ae70-0e9edb402458-kube-api-access-x7b97" pod "4c58df41-54f0-4939-ae70-0e9edb402458" (UID: "4c58df41-54f0-4939-ae70-0e9edb402458"). InnerVolumeSpecName "kube-api-access-x7b97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:04:31.768156 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:31.768120 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:31.839627 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:31.839552 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7b97\" (UniqueName: \"kubernetes.io/projected/4c58df41-54f0-4939-ae70-0e9edb402458-kube-api-access-x7b97\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:31.896209 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:31.896171 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:32.020382 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:32.020352 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:32.114745 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:32.114713 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7"] Nov 04 05:04:32.123636 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:32.123610 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7"] Nov 04 05:04:33.230999 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:33.230957 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c58df41-54f0-4939-ae70-0e9edb402458" path="/var/lib/kubelet/pods/4c58df41-54f0-4939-ae70-0e9edb402458/volumes" Nov 04 05:04:33.724592 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:33.724561 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:33.861571 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:33.861537 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:36.094584 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.094551 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl"] Nov 04 05:04:36.095226 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.095207 4581 cpu_manager.go:414] "containerMap: removing container" podUID="aa8dcd7f-0f9f-4eef-a831-8d164c92830e" containerName="istio-proxy" Nov 04 05:04:36.095296 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.095227 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa8dcd7f-0f9f-4eef-a831-8d164c92830e" containerName="istio-proxy" Nov 04 05:04:36.095296 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.095241 4581 cpu_manager.go:414] "containerMap: removing container" podUID="aa8dcd7f-0f9f-4eef-a831-8d164c92830e" containerName="istio-proxy" Nov 04 05:04:36.095296 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.095246 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa8dcd7f-0f9f-4eef-a831-8d164c92830e" containerName="istio-proxy" Nov 04 05:04:36.095296 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.095255 4581 cpu_manager.go:414] "containerMap: removing container" podUID="510d29ba-d057-4f6b-af44-aa85ab0c4ed6" containerName="istio-proxy" Nov 04 05:04:36.095296 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.095260 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510d29ba-d057-4f6b-af44-aa85ab0c4ed6" containerName="istio-proxy" Nov 04 05:04:36.095296 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.095270 4581 cpu_manager.go:414] "containerMap: removing container" podUID="4c58df41-54f0-4939-ae70-0e9edb402458" containerName="payload-processing" Nov 04 05:04:36.095296 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.095274 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c58df41-54f0-4939-ae70-0e9edb402458" containerName="payload-processing" Nov 04 05:04:36.095625 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.095307 4581 cpu_manager.go:414] "containerMap: removing container" podUID="1c7c486a-863f-49ff-abb3-819e3905e640" containerName="main" Nov 04 05:04:36.095625 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.095311 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7c486a-863f-49ff-abb3-819e3905e640" containerName="main" Nov 04 05:04:36.095625 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.095391 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa8dcd7f-0f9f-4eef-a831-8d164c92830e" containerName="istio-proxy" Nov 04 05:04:36.095625 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.095397 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="510d29ba-d057-4f6b-af44-aa85ab0c4ed6" containerName="istio-proxy" Nov 04 05:04:36.095625 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.095404 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c58df41-54f0-4939-ae70-0e9edb402458" containerName="payload-processing" Nov 04 05:04:36.095625 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.095415 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7c486a-863f-49ff-abb3-819e3905e640" containerName="main" Nov 04 05:04:36.095625 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.095423 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa8dcd7f-0f9f-4eef-a831-8d164c92830e" containerName="istio-proxy" Nov 04 05:04:36.100091 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.100069 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:04:36.176777 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.176744 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:36.185849 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.185820 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-dshm\") pod \"distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl\" (UID: \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:04:36.186022 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.185863 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-tls-certs\") pod \"distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl\" (UID: \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:04:36.186022 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.185949 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-tmp-dir\") pod \"distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl\" (UID: \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:04:36.186178 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.186019 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hvtc\" (UniqueName: \"kubernetes.io/projected/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-kube-api-access-5hvtc\") pod \"distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl\" (UID: \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:04:36.186178 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.186084 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-model-cache\") pod \"distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl\" (UID: \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:04:36.186308 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.186285 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-home\") pod \"distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl\" (UID: \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:04:36.204154 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.204133 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-4cce33\"/\"distinct-2-w5-4cce33-kserve-self-signed-certs\"" Nov 04 05:04:36.206801 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.206783 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl"] Nov 04 05:04:36.287564 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.287535 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-tls-certs\") pod \"distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl\" (UID: \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:04:36.287727 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.287580 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-tmp-dir\") pod \"distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl\" (UID: \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:04:36.287727 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.287614 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hvtc\" (UniqueName: \"kubernetes.io/projected/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-kube-api-access-5hvtc\") pod \"distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl\" (UID: \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:04:36.287727 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.287667 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-model-cache\") pod \"distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl\" (UID: \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:04:36.287894 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.287777 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-home\") pod \"distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl\" (UID: \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:04:36.287894 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.287832 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-dshm\") pod \"distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl\" (UID: \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:04:36.288024 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.287927 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-tmp-dir\") pod \"distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl\" (UID: \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:04:36.288146 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.288113 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-model-cache\") pod \"distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl\" (UID: \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:04:36.288262 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.288161 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-home\") pod \"distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl\" (UID: \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:04:36.290081 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.290062 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-dshm\") pod \"distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl\" (UID: \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:04:36.290340 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.290323 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-tls-certs\") pod \"distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl\" (UID: \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:04:36.345136 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.345082 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hvtc\" (UniqueName: \"kubernetes.io/projected/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-kube-api-access-5hvtc\") pod \"distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl\" (UID: \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:04:36.411716 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.411685 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:04:36.642350 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.642320 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:36.789667 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:36.789638 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl"] Nov 04 05:04:36.791450 ip-10-0-141-132 kubenswrapper[4581]: W1104 05:04:36.791424 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4977f3bb_f6f9_4fc1_9934_9e02d5c2e937.slice/crio-b261adb8c3b5be97e6575fb82ac2a2355e1fcff1c1a6c73e4448efaf869d2278 WatchSource:0}: Error finding container b261adb8c3b5be97e6575fb82ac2a2355e1fcff1c1a6c73e4448efaf869d2278: Status 404 returned error can't find the container with id b261adb8c3b5be97e6575fb82ac2a2355e1fcff1c1a6c73e4448efaf869d2278 Nov 04 05:04:37.048800 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.048712 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56"] Nov 04 05:04:37.052546 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.052523 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.062287 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.062268 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq"] Nov 04 05:04:37.066494 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.066478 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.111851 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.111819 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56"] Nov 04 05:04:37.178572 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.178539 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq"] Nov 04 05:04:37.197829 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.197801 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-envoy\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.198019 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.197834 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-envoy\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.198019 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.197866 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-token\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.198154 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.198055 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-credential-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.198154 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.198096 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnfgc\" (UniqueName: \"kubernetes.io/projected/1ce6a777-2cfb-4cf5-905a-209d8d82f331-kube-api-access-qnfgc\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.198221 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.198186 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istiod-ca-cert\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.198256 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.198221 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-workload-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.198256 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.198239 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-data\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.198375 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.198360 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-podinfo\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.198420 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.198409 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-podinfo\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.198501 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.198477 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-workload-certs\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.198549 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.198536 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-workload-certs\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.198585 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.198559 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-data\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.198628 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.198581 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-token\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.198686 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.198665 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istiod-ca-cert\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.198747 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.198730 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-credential-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.198783 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.198760 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf2gv\" (UniqueName: \"kubernetes.io/projected/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-kube-api-access-lf2gv\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.198826 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.198809 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-workload-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.300071 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.299974 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-podinfo\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.300278 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.300253 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-workload-certs\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.300401 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.300281 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-workload-certs\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.300401 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.300321 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-data\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.300401 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.300359 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-token\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.300401 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.300390 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istiod-ca-cert\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.300626 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.300448 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-credential-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.300626 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.300473 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf2gv\" (UniqueName: \"kubernetes.io/projected/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-kube-api-access-lf2gv\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.300626 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.300532 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-workload-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.300626 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.300619 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-envoy\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.300828 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.300651 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-envoy\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.300828 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.300682 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-token\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.300828 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.300690 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-workload-certs\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.300828 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.300697 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-data\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.300828 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.300615 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-workload-certs\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.301114 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.300858 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-workload-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.301114 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.300954 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-credential-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.301114 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.300964 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-credential-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.301114 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.301035 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnfgc\" (UniqueName: \"kubernetes.io/projected/1ce6a777-2cfb-4cf5-905a-209d8d82f331-kube-api-access-qnfgc\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.301114 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.301083 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istiod-ca-cert\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.301384 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.301125 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-workload-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.301384 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.301153 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-data\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.301384 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.301244 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-podinfo\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.301539 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.301394 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istiod-ca-cert\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.301539 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.301427 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-data\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.301649 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.301624 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-credential-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.301914 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.301888 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-workload-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.302014 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.301955 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istiod-ca-cert\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.303157 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.303136 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-podinfo\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.303616 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.303592 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-envoy\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.303616 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.303590 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-envoy\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.303732 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.303663 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-podinfo\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.357900 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.357863 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-token\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.358041 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.357874 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-token\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.358087 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.358038 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnfgc\" (UniqueName: \"kubernetes.io/projected/1ce6a777-2cfb-4cf5-905a-209d8d82f331-kube-api-access-qnfgc\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.358087 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.358052 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf2gv\" (UniqueName: \"kubernetes.io/projected/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-kube-api-access-lf2gv\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.365788 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.365771 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:37.380057 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.380034 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:37.730252 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.730216 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" event={"ID":"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937","Type":"ContainerStarted","Data":"bcf5f2ee14e657c11c4eef6e4544329ae23a3333494f7db7c49e9088d96d8a9a"} Nov 04 05:04:37.730252 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.730264 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" event={"ID":"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937","Type":"ContainerStarted","Data":"b261adb8c3b5be97e6575fb82ac2a2355e1fcff1c1a6c73e4448efaf869d2278"} Nov 04 05:04:37.801091 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.801058 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56"] Nov 04 05:04:37.802308 ip-10-0-141-132 kubenswrapper[4581]: W1104 05:04:37.802279 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce6a777_2cfb_4cf5_905a_209d8d82f331.slice/crio-401095da52b231c3c81d29120e1b9cceffc84dda9c0d59db40276af8d05621c5 WatchSource:0}: Error finding container 401095da52b231c3c81d29120e1b9cceffc84dda9c0d59db40276af8d05621c5: Status 404 returned error can't find the container with id 401095da52b231c3c81d29120e1b9cceffc84dda9c0d59db40276af8d05621c5 Nov 04 05:04:37.803867 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.803842 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq"] Nov 04 05:04:37.804696 ip-10-0-141-132 kubenswrapper[4581]: W1104 05:04:37.804671 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed81f25_e6fc_4a0c_a631_3ac69d7d64d2.slice/crio-595fa9326914411f02983b65774a1957077b19dafee548a0860351057aa06018 WatchSource:0}: Error finding container 595fa9326914411f02983b65774a1957077b19dafee548a0860351057aa06018: Status 404 returned error can't find the container with id 595fa9326914411f02983b65774a1957077b19dafee548a0860351057aa06018 Nov 04 05:04:37.806263 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.806230 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:04:37.806367 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.806295 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:04:37.806367 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.806338 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:04:37.806771 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.806733 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:04:37.806803 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.806787 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:04:37.806882 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.806816 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:04:37.848748 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.848707 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" podStartSLOduration=1.8486929220000001 podStartE2EDuration="1.848692922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:04:37.845853493 +0000 UTC m=+1036.736710514" watchObservedRunningTime="2026-11-04 05:04:37.848692922 +0000 UTC m=+1036.739549942" Nov 04 05:04:37.876882 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:37.876852 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:38.412457 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:38.412425 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:04:38.737767 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:38.737670 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" event={"ID":"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2","Type":"ContainerStarted","Data":"05f600a8213dcdb5c37fac19e48c3680a01464c59d9205f63bfd30d195dd93cf"} Nov 04 05:04:38.737767 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:38.737735 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" event={"ID":"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2","Type":"ContainerStarted","Data":"595fa9326914411f02983b65774a1957077b19dafee548a0860351057aa06018"} Nov 04 05:04:38.739377 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:38.739350 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" event={"ID":"1ce6a777-2cfb-4cf5-905a-209d8d82f331","Type":"ContainerStarted","Data":"4a42851e8dafb92187ca246ee09c34582945257d023a7421cde6c7359cb206e2"} Nov 04 05:04:38.739504 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:38.739398 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" event={"ID":"1ce6a777-2cfb-4cf5-905a-209d8d82f331","Type":"ContainerStarted","Data":"401095da52b231c3c81d29120e1b9cceffc84dda9c0d59db40276af8d05621c5"} Nov 04 05:04:39.004071 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:39.003208 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" podStartSLOduration=2.00319079 podStartE2EDuration="2.00319079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:04:38.875391791 +0000 UTC m=+1037.766248815" watchObservedRunningTime="2026-11-04 05:04:39.00319079 +0000 UTC m=+1037.894047812" Nov 04 05:04:39.373014 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:39.367171 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:39.373014 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:39.369364 4581 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 04 05:04:39.373014 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:39.369432 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" podUID="1ce6a777-2cfb-4cf5-905a-209d8d82f331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 04 05:04:39.391207 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:39.390602 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:39.391207 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:39.390750 4581 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 04 05:04:39.391207 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:39.390792 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" podUID="2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 04 05:04:40.374014 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:40.370385 4581 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 04 05:04:40.374014 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:40.370460 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" podUID="1ce6a777-2cfb-4cf5-905a-209d8d82f331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 04 05:04:40.390429 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:40.390385 4581 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 04 05:04:40.390632 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:40.390455 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" podUID="2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 04 05:04:40.620707 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:40.620410 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:40.747523 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:40.747418 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:40.908923 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:40.908869 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" podStartSLOduration=3.908850709 podStartE2EDuration="3.908850709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:04:39.003482838 +0000 UTC m=+1037.894339858" watchObservedRunningTime="2026-11-04 05:04:40.908850709 +0000 UTC m=+1039.799707728" Nov 04 05:04:40.910313 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:40.910290 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp"] Nov 04 05:04:40.910557 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:40.910515 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" podUID="6ec52444-399d-4a73-9652-ae7d3f46621d" containerName="maas-api" containerID="cri-o://1c9cca6d063275d2b734aa1d1d2a51422034f8da5f7794490b705c8c4207ebba" gracePeriod=30 Nov 04 05:04:41.015774 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.015691 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:41.398452 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.398415 4581 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 04 05:04:41.407519 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.407459 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" podUID="1ce6a777-2cfb-4cf5-905a-209d8d82f331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 04 05:04:41.408410 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.408386 4581 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 04 05:04:41.408639 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.408605 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" podUID="2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 04 05:04:41.418022 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.416778 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" Nov 04 05:04:41.419219 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.419200 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:41.509347 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.509310 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ec52444-399d-4a73-9652-ae7d3f46621d-kube-api-access-hnr55\" (UniqueName: \"kubernetes.io/projected/6ec52444-399d-4a73-9652-ae7d3f46621d-kube-api-access-hnr55\") pod \"6ec52444-399d-4a73-9652-ae7d3f46621d\" (UID: \"6ec52444-399d-4a73-9652-ae7d3f46621d\") " Nov 04 05:04:41.509520 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.509416 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ec52444-399d-4a73-9652-ae7d3f46621d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ec52444-399d-4a73-9652-ae7d3f46621d-maas-api-tls\") pod \"6ec52444-399d-4a73-9652-ae7d3f46621d\" (UID: \"6ec52444-399d-4a73-9652-ae7d3f46621d\") " Nov 04 05:04:41.509611 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.509542 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ec52444-399d-4a73-9652-ae7d3f46621d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec52444-399d-4a73-9652-ae7d3f46621d-metrics-certs\") pod \"6ec52444-399d-4a73-9652-ae7d3f46621d\" (UID: \"6ec52444-399d-4a73-9652-ae7d3f46621d\") " Nov 04 05:04:41.509673 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.509611 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ec52444-399d-4a73-9652-ae7d3f46621d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec52444-399d-4a73-9652-ae7d3f46621d-openshift-service-ca\") pod \"6ec52444-399d-4a73-9652-ae7d3f46621d\" (UID: \"6ec52444-399d-4a73-9652-ae7d3f46621d\") " Nov 04 05:04:41.510461 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.510420 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec52444-399d-4a73-9652-ae7d3f46621d-openshift-service-ca" pod "6ec52444-399d-4a73-9652-ae7d3f46621d" (UID: "6ec52444-399d-4a73-9652-ae7d3f46621d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:04:41.512637 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.512617 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ec52444-399d-4a73-9652-ae7d3f46621d-maas-api-tls" pod "6ec52444-399d-4a73-9652-ae7d3f46621d" (UID: "6ec52444-399d-4a73-9652-ae7d3f46621d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:04:41.512721 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.512657 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ec52444-399d-4a73-9652-ae7d3f46621d-metrics-certs" pod "6ec52444-399d-4a73-9652-ae7d3f46621d" (UID: "6ec52444-399d-4a73-9652-ae7d3f46621d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:04:41.513432 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.513408 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec52444-399d-4a73-9652-ae7d3f46621d-kube-api-access-hnr55" pod "6ec52444-399d-4a73-9652-ae7d3f46621d" (UID: "6ec52444-399d-4a73-9652-ae7d3f46621d"). InnerVolumeSpecName "kube-api-access-hnr55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:04:41.612856 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.612821 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnr55\" (UniqueName: \"kubernetes.io/projected/6ec52444-399d-4a73-9652-ae7d3f46621d-kube-api-access-hnr55\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:41.612856 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.612857 4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ec52444-399d-4a73-9652-ae7d3f46621d-maas-api-tls\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:41.613167 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.612872 4581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec52444-399d-4a73-9652-ae7d3f46621d-metrics-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:41.613167 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.612887 4581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec52444-399d-4a73-9652-ae7d3f46621d-openshift-service-ca\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:41.783010 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.782754 4581 generic.go:356] "Generic (PLEG): container finished" podID="6ec52444-399d-4a73-9652-ae7d3f46621d" containerID="1c9cca6d063275d2b734aa1d1d2a51422034f8da5f7794490b705c8c4207ebba" exitCode=0 Nov 04 05:04:41.783010 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.782874 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" event={"ID":"6ec52444-399d-4a73-9652-ae7d3f46621d","Type":"ContainerDied","Data":"1c9cca6d063275d2b734aa1d1d2a51422034f8da5f7794490b705c8c4207ebba"} Nov 04 05:04:41.783010 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.782924 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" event={"ID":"6ec52444-399d-4a73-9652-ae7d3f46621d","Type":"ContainerDied","Data":"d701e31dfce2a780ae43076ed4383634c5a83d1ada218dccc52801c0e3d4f156"} Nov 04 05:04:41.783010 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.782955 4581 scope.go:122] "RemoveContainer" containerID="1c9cca6d063275d2b734aa1d1d2a51422034f8da5f7794490b705c8c4207ebba" Nov 04 05:04:41.783377 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.783147 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp" Nov 04 05:04:41.821186 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.821163 4581 scope.go:122] "RemoveContainer" containerID="1c9cca6d063275d2b734aa1d1d2a51422034f8da5f7794490b705c8c4207ebba" Nov 04 05:04:41.821514 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:04:41.821493 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c9cca6d063275d2b734aa1d1d2a51422034f8da5f7794490b705c8c4207ebba\": container with ID starting with 1c9cca6d063275d2b734aa1d1d2a51422034f8da5f7794490b705c8c4207ebba not found: ID does not exist" containerID="1c9cca6d063275d2b734aa1d1d2a51422034f8da5f7794490b705c8c4207ebba" Nov 04 05:04:41.821570 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.821524 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c9cca6d063275d2b734aa1d1d2a51422034f8da5f7794490b705c8c4207ebba"} err="failed to get container status \"1c9cca6d063275d2b734aa1d1d2a51422034f8da5f7794490b705c8c4207ebba\": rpc error: code = NotFound desc = could not find container \"1c9cca6d063275d2b734aa1d1d2a51422034f8da5f7794490b705c8c4207ebba\": container with ID starting with 1c9cca6d063275d2b734aa1d1d2a51422034f8da5f7794490b705c8c4207ebba not found: ID does not exist" Nov 04 05:04:41.991215 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:41.991055 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp"] Nov 04 05:04:42.002886 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:42.002846 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1874d8-75dbc9c476-j7pfp"] Nov 04 05:04:42.378012 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:42.369407 4581 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 04 05:04:42.378012 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:42.369466 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" podUID="1ce6a777-2cfb-4cf5-905a-209d8d82f331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 04 05:04:42.389474 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:42.389353 4581 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 04 05:04:42.389474 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:42.389416 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" podUID="2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 04 05:04:42.942330 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:42.942292 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:43.221128 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:43.221054 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:43.231449 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:43.231422 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ec52444-399d-4a73-9652-ae7d3f46621d" path="/var/lib/kubelet/pods/6ec52444-399d-4a73-9652-ae7d3f46621d/volumes" Nov 04 05:04:43.345221 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:43.345131 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:43.366477 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:43.366445 4581 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 04 05:04:43.366781 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:43.366742 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" podUID="1ce6a777-2cfb-4cf5-905a-209d8d82f331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 04 05:04:43.384697 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:43.384588 4581 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 04 05:04:43.384697 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:43.384638 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" podUID="2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 04 05:04:44.366719 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:44.366681 4581 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 04 05:04:44.367207 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:44.366751 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" podUID="1ce6a777-2cfb-4cf5-905a-209d8d82f331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 04 05:04:44.381357 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:44.381324 4581 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 04 05:04:44.381490 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:44.381388 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" podUID="2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 04 05:04:44.406120 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:44.406087 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:44.575416 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:44.575376 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:44.747201 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:44.747106 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:45.268142 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:45.268100 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm"] Nov 04 05:04:45.269211 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:45.269178 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" podUID="ad5ae17b-aca8-4988-8ed1-7042c62fc32f" containerName="istio-proxy" containerID="cri-o://19e7ca7a1865fad06fc199a041c373515351a514984b767be366b1b52bf20b32" gracePeriod=30 Nov 04 05:04:45.291305 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:45.291274 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:45.459906 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:45.459875 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:45.461324 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:45.461124 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:45.461885 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:45.461866 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:04:45.482061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:45.482037 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:45.482371 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:45.482349 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:45.483234 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:45.483216 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:04:46.392718 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:46.392684 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb"] Nov 04 05:04:46.412215 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:46.412192 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:04:46.427273 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:46.427250 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:04:46.484726 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:46.484688 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:46.874503 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:46.874463 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:04:47.067631 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:47.067597 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:47.887358 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:04:47.887320 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 05:04:48.166179 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:48.166089 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:49.236320 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:49.236288 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:49.892826 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:49.892785 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:50.023159 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.023125 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:50.566629 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.566563 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 05:04:50.730364 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.730338 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istiod-ca-cert\") pod \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " Nov 04 05:04:50.730553 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.730380 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-kube-api-access-4b8cq\" (UniqueName: \"kubernetes.io/projected/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-kube-api-access-4b8cq\") pod \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " Nov 04 05:04:50.730553 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.730445 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-workload-socket\") pod \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " Nov 04 05:04:50.730553 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.730463 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-podinfo\") pod \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " Nov 04 05:04:50.730553 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.730481 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-envoy\") pod \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " Nov 04 05:04:50.730553 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.730520 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-credential-socket\") pod \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " Nov 04 05:04:50.730856 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.730598 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-workload-certs\") pod \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " Nov 04 05:04:50.730856 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.730668 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-data\") pod \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " Nov 04 05:04:50.730856 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.730760 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-token\" (UniqueName: \"kubernetes.io/projected/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-token\") pod \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\" (UID: \"ad5ae17b-aca8-4988-8ed1-7042c62fc32f\") " Nov 04 05:04:50.730856 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.730771 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-workload-socket" pod "ad5ae17b-aca8-4988-8ed1-7042c62fc32f" (UID: "ad5ae17b-aca8-4988-8ed1-7042c62fc32f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:50.731086 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.730893 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istiod-ca-cert" pod "ad5ae17b-aca8-4988-8ed1-7042c62fc32f" (UID: "ad5ae17b-aca8-4988-8ed1-7042c62fc32f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:04:50.731086 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.730915 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-workload-certs" pod "ad5ae17b-aca8-4988-8ed1-7042c62fc32f" (UID: "ad5ae17b-aca8-4988-8ed1-7042c62fc32f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:50.731086 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.730929 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-credential-socket" pod "ad5ae17b-aca8-4988-8ed1-7042c62fc32f" (UID: "ad5ae17b-aca8-4988-8ed1-7042c62fc32f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:50.731086 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.731071 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-data" pod "ad5ae17b-aca8-4988-8ed1-7042c62fc32f" (UID: "ad5ae17b-aca8-4988-8ed1-7042c62fc32f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:50.731402 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.731341 4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-data\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:50.731562 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.731478 4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istiod-ca-cert\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:50.731562 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.731542 4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-workload-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:50.731562 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.731556 4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-credential-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:50.731562 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.731586 4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-workload-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:50.733016 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.732961 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-kube-api-access-4b8cq" pod "ad5ae17b-aca8-4988-8ed1-7042c62fc32f" (UID: "ad5ae17b-aca8-4988-8ed1-7042c62fc32f"). InnerVolumeSpecName "kube-api-access-4b8cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:04:50.733117 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.733065 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-token" pod "ad5ae17b-aca8-4988-8ed1-7042c62fc32f" (UID: "ad5ae17b-aca8-4988-8ed1-7042c62fc32f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:04:50.733521 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.733496 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-podinfo" pod "ad5ae17b-aca8-4988-8ed1-7042c62fc32f" (UID: "ad5ae17b-aca8-4988-8ed1-7042c62fc32f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:04:50.733633 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.733529 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-envoy" pod "ad5ae17b-aca8-4988-8ed1-7042c62fc32f" (UID: "ad5ae17b-aca8-4988-8ed1-7042c62fc32f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:50.832892 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.832816 4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-token\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:50.832892 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.832845 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b8cq\" (UniqueName: \"kubernetes.io/projected/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-kube-api-access-4b8cq\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:50.832892 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.832876 4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-podinfo\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:50.832892 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.832885 4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae17b-aca8-4988-8ed1-7042c62fc32f-istio-envoy\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:50.886085 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.886048 4581 generic.go:356] "Generic (PLEG): container finished" podID="ad5ae17b-aca8-4988-8ed1-7042c62fc32f" containerID="19e7ca7a1865fad06fc199a041c373515351a514984b767be366b1b52bf20b32" exitCode=0 Nov 04 05:04:50.886247 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.886104 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" Nov 04 05:04:50.886247 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.886132 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" event={"ID":"ad5ae17b-aca8-4988-8ed1-7042c62fc32f","Type":"ContainerDied","Data":"19e7ca7a1865fad06fc199a041c373515351a514984b767be366b1b52bf20b32"} Nov 04 05:04:50.886247 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.886189 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm" event={"ID":"ad5ae17b-aca8-4988-8ed1-7042c62fc32f","Type":"ContainerDied","Data":"aa5fb2aaf2855ecc810c13534fd0aaa97f2e9581eae91c0a0b4b4b1407bc1cf1"} Nov 04 05:04:50.886247 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.886213 4581 scope.go:122] "RemoveContainer" containerID="19e7ca7a1865fad06fc199a041c373515351a514984b767be366b1b52bf20b32" Nov 04 05:04:50.897671 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.897638 4581 scope.go:122] "RemoveContainer" containerID="8098505851b03a6fe65a0014515ea965c1f6e89ce749aa093d59401232c0d2dc" Nov 04 05:04:50.908896 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.908875 4581 scope.go:122] "RemoveContainer" containerID="19e7ca7a1865fad06fc199a041c373515351a514984b767be366b1b52bf20b32" Nov 04 05:04:50.909252 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:04:50.909231 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19e7ca7a1865fad06fc199a041c373515351a514984b767be366b1b52bf20b32\": container with ID starting with 19e7ca7a1865fad06fc199a041c373515351a514984b767be366b1b52bf20b32 not found: ID does not exist" containerID="19e7ca7a1865fad06fc199a041c373515351a514984b767be366b1b52bf20b32" Nov 04 05:04:50.909338 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.909269 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e7ca7a1865fad06fc199a041c373515351a514984b767be366b1b52bf20b32"} err="failed to get container status \"19e7ca7a1865fad06fc199a041c373515351a514984b767be366b1b52bf20b32\": rpc error: code = NotFound desc = could not find container \"19e7ca7a1865fad06fc199a041c373515351a514984b767be366b1b52bf20b32\": container with ID starting with 19e7ca7a1865fad06fc199a041c373515351a514984b767be366b1b52bf20b32 not found: ID does not exist" Nov 04 05:04:50.909338 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.909289 4581 scope.go:122] "RemoveContainer" containerID="8098505851b03a6fe65a0014515ea965c1f6e89ce749aa093d59401232c0d2dc" Nov 04 05:04:50.909587 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:04:50.909569 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8098505851b03a6fe65a0014515ea965c1f6e89ce749aa093d59401232c0d2dc\": container with ID starting with 8098505851b03a6fe65a0014515ea965c1f6e89ce749aa093d59401232c0d2dc not found: ID does not exist" containerID="8098505851b03a6fe65a0014515ea965c1f6e89ce749aa093d59401232c0d2dc" Nov 04 05:04:50.909646 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.909599 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8098505851b03a6fe65a0014515ea965c1f6e89ce749aa093d59401232c0d2dc"} err="failed to get container status \"8098505851b03a6fe65a0014515ea965c1f6e89ce749aa093d59401232c0d2dc\": rpc error: code = NotFound desc = could not find container \"8098505851b03a6fe65a0014515ea965c1f6e89ce749aa093d59401232c0d2dc\": container with ID starting with 8098505851b03a6fe65a0014515ea965c1f6e89ce749aa093d59401232c0d2dc not found: ID does not exist" Nov 04 05:04:50.955669 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:50.955627 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:51.055222 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:51.055190 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm"] Nov 04 05:04:51.058491 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:51.058462 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm"] Nov 04 05:04:51.230546 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:51.230517 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad5ae17b-aca8-4988-8ed1-7042c62fc32f" path="/var/lib/kubelet/pods/ad5ae17b-aca8-4988-8ed1-7042c62fc32f/volumes" Nov 04 05:04:52.652130 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:52.652093 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:52.772519 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:52.772483 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:53.033072 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:53.032958 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:53.150540 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:53.150501 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:54.206220 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:54.206189 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7"] Nov 04 05:04:54.256437 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:54.256401 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd"] Nov 04 05:04:54.362835 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:54.362802 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:57.828319 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:57.828285 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:58.816553 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:58.816516 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:59.475171 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:59.475133 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:04:59.657455 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:04:59.657416 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:00.609756 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:00.609720 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:01.419165 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.419123 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" podUID="7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674" containerName="main" containerID="cri-o://a7ccc60144f0e802d0949d8358f006a12a39dcfa0dcbc8099fae005653d0bce5" gracePeriod=45 Nov 04 05:05:01.684390 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.684327 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:05:01.754785 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.754750 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-home\" (UniqueName: \"kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-home\") pod \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\" (UID: \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\") " Nov 04 05:05:01.754978 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.754825 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-tmp-dir\") pod \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\" (UID: \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\") " Nov 04 05:05:01.754978 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.754855 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-model-cache\") pod \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\" (UID: \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\") " Nov 04 05:05:01.754978 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.754937 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-dshm\") pod \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\" (UID: \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\") " Nov 04 05:05:01.755169 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.755053 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-kube-api-access-wsvvd\" (UniqueName: \"kubernetes.io/projected/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-kube-api-access-wsvvd\") pod \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\" (UID: \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\") " Nov 04 05:05:01.755169 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.755100 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-home" pod "7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674" (UID: "7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:01.755169 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.755117 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-tmp-dir" pod "7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674" (UID: "7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:01.755169 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.755135 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-tls-certs\" (UniqueName: \"kubernetes.io/secret/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-tls-certs\") pod \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\" (UID: \"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674\") " Nov 04 05:05:01.755380 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.755217 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-model-cache" pod "7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674" (UID: "7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:01.755515 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.755496 4581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-tmp-dir\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:01.755567 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.755521 4581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-model-cache\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:01.755567 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.755534 4581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-home\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:01.757432 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.757407 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-dshm" pod "7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674" (UID: "7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:01.757545 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.757435 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-kube-api-access-wsvvd" pod "7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674" (UID: "7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674"). InnerVolumeSpecName "kube-api-access-wsvvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:01.757545 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.757431 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-tls-certs" pod "7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674" (UID: "7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:05:01.856300 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.856268 4581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-dshm\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:01.856300 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.856298 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsvvd\" (UniqueName: \"kubernetes.io/projected/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-kube-api-access-wsvvd\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:01.856502 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.856311 4581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674-tls-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:01.957588 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.957499 4581 generic.go:356] "Generic (PLEG): container finished" podID="7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674" containerID="a7ccc60144f0e802d0949d8358f006a12a39dcfa0dcbc8099fae005653d0bce5" exitCode=0 Nov 04 05:05:01.957588 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.957539 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" event={"ID":"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674","Type":"ContainerDied","Data":"a7ccc60144f0e802d0949d8358f006a12a39dcfa0dcbc8099fae005653d0bce5"} Nov 04 05:05:01.957588 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.957590 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" event={"ID":"7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674","Type":"ContainerDied","Data":"002e88e8071a35a8201cd235316ba46eb8bbc1707640d4301bda276824f21bd3"} Nov 04 05:05:01.957896 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.957611 4581 scope.go:122] "RemoveContainer" containerID="a7ccc60144f0e802d0949d8358f006a12a39dcfa0dcbc8099fae005653d0bce5" Nov 04 05:05:01.957896 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.957553 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb" Nov 04 05:05:01.967874 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.967852 4581 scope.go:122] "RemoveContainer" containerID="a7ccc60144f0e802d0949d8358f006a12a39dcfa0dcbc8099fae005653d0bce5" Nov 04 05:05:01.968223 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:05:01.968198 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7ccc60144f0e802d0949d8358f006a12a39dcfa0dcbc8099fae005653d0bce5\": container with ID starting with a7ccc60144f0e802d0949d8358f006a12a39dcfa0dcbc8099fae005653d0bce5 not found: ID does not exist" containerID="a7ccc60144f0e802d0949d8358f006a12a39dcfa0dcbc8099fae005653d0bce5" Nov 04 05:05:01.968332 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:01.968230 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7ccc60144f0e802d0949d8358f006a12a39dcfa0dcbc8099fae005653d0bce5"} err="failed to get container status \"a7ccc60144f0e802d0949d8358f006a12a39dcfa0dcbc8099fae005653d0bce5\": rpc error: code = NotFound desc = could not find container \"a7ccc60144f0e802d0949d8358f006a12a39dcfa0dcbc8099fae005653d0bce5\": container with ID starting with a7ccc60144f0e802d0949d8358f006a12a39dcfa0dcbc8099fae005653d0bce5 not found: ID does not exist" Nov 04 05:05:02.108628 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:02.108589 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:02.122183 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:02.122150 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb"] Nov 04 05:05:02.125785 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:02.125757 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-370078/simulator-w2-370078-kserve-5b84dbd9f4-c8bjb"] Nov 04 05:05:02.811327 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:02.811288 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:03.230700 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:03.230670 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674" path="/var/lib/kubelet/pods/7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674/volumes" Nov 04 05:05:04.311595 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:04.311549 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:04.444713 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:04.444680 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:04.862809 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:04.862767 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:05.059091 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:05.059060 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:06.390222 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.390187 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6"] Nov 04 05:05:06.390670 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.390657 4581 cpu_manager.go:414] "containerMap: removing container" podUID="ad5ae17b-aca8-4988-8ed1-7042c62fc32f" containerName="istio-proxy" Nov 04 05:05:06.390725 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.390671 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad5ae17b-aca8-4988-8ed1-7042c62fc32f" containerName="istio-proxy" Nov 04 05:05:06.390725 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.390685 4581 cpu_manager.go:414] "containerMap: removing container" podUID="6ec52444-399d-4a73-9652-ae7d3f46621d" containerName="maas-api" Nov 04 05:05:06.390725 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.390690 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec52444-399d-4a73-9652-ae7d3f46621d" containerName="maas-api" Nov 04 05:05:06.390725 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.390714 4581 cpu_manager.go:414] "containerMap: removing container" podUID="7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674" containerName="main" Nov 04 05:05:06.390725 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.390719 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674" containerName="main" Nov 04 05:05:06.390872 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.390787 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec52444-399d-4a73-9652-ae7d3f46621d" containerName="maas-api" Nov 04 05:05:06.390872 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.390797 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad5ae17b-aca8-4988-8ed1-7042c62fc32f" containerName="istio-proxy" Nov 04 05:05:06.390872 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.390805 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a196cc6-b6bd-4fb1-8fb7-9588bb4ff674" containerName="main" Nov 04 05:05:06.395204 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.395186 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.453304 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.453270 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6"] Nov 04 05:05:06.500045 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.499981 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-4cce33-openshift-default-dockercfg-qrlrc\"" Nov 04 05:05:06.511542 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.511493 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-istio-envoy\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.511719 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.511621 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-credential-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.511719 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.511654 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-workload-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.511859 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.511748 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-workload-certs\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.511859 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.511827 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-istio-data\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.511859 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.511852 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b652ed62-2de7-4888-91e1-686213b028f3-istio-token\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.512063 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.511929 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b652ed62-2de7-4888-91e1-686213b028f3-istiod-ca-cert\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.512063 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.512013 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdwx7\" (UniqueName: \"kubernetes.io/projected/b652ed62-2de7-4888-91e1-686213b028f3-kube-api-access-kdwx7\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.512189 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.512071 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b652ed62-2de7-4888-91e1-686213b028f3-istio-podinfo\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.613264 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.613228 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-workload-certs\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.613456 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.613291 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-istio-data\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.613456 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.613316 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b652ed62-2de7-4888-91e1-686213b028f3-istio-token\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.613456 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.613353 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b652ed62-2de7-4888-91e1-686213b028f3-istiod-ca-cert\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.613456 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.613448 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdwx7\" (UniqueName: \"kubernetes.io/projected/b652ed62-2de7-4888-91e1-686213b028f3-kube-api-access-kdwx7\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.613665 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.613481 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b652ed62-2de7-4888-91e1-686213b028f3-istio-podinfo\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.613665 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.613625 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-istio-envoy\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.613773 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.613677 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-workload-certs\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.613773 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.613711 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-credential-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.613773 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.613748 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-workload-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.613773 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.613754 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-istio-data\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.614124 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.614094 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-credential-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.614218 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.614095 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-workload-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.614218 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.614206 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b652ed62-2de7-4888-91e1-686213b028f3-istiod-ca-cert\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.615931 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.615899 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-istio-envoy\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.616168 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.616152 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b652ed62-2de7-4888-91e1-686213b028f3-istio-podinfo\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.670973 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.670898 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b652ed62-2de7-4888-91e1-686213b028f3-istio-token\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.671132 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.671048 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdwx7\" (UniqueName: \"kubernetes.io/projected/b652ed62-2de7-4888-91e1-686213b028f3-kube-api-access-kdwx7\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.708500 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.708471 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:06.738690 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:06.738657 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:07.130567 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:07.130535 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:07.139062 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:07.139037 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6"] Nov 04 05:05:07.141199 ip-10-0-141-132 kubenswrapper[4581]: W1104 05:05:07.141174 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb652ed62_2de7_4888_91e1_686213b028f3.slice/crio-db088cce490240018f1d12aecf6349a2bcf52c108f35eb2785ca709d56b4cc12 WatchSource:0}: Error finding container db088cce490240018f1d12aecf6349a2bcf52c108f35eb2785ca709d56b4cc12: Status 404 returned error can't find the container with id db088cce490240018f1d12aecf6349a2bcf52c108f35eb2785ca709d56b4cc12 Nov 04 05:05:07.143664 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:07.143633 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:05:07.143768 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:07.143712 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:05:07.143768 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:07.143760 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:05:07.581616 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:07.581524 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:07.708719 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:07.708683 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:07.995230 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:07.995191 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" event={"ID":"b652ed62-2de7-4888-91e1-686213b028f3","Type":"ContainerStarted","Data":"d0bd0db28c6f21c1df6800eb9c2bfcb0be95a1bd66d521762f6a3d1f808911f8"} Nov 04 05:05:07.995447 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:07.995277 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" event={"ID":"b652ed62-2de7-4888-91e1-686213b028f3","Type":"ContainerStarted","Data":"db088cce490240018f1d12aecf6349a2bcf52c108f35eb2785ca709d56b4cc12"} Nov 04 05:05:08.113556 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:08.113461 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" podStartSLOduration=2.113387669 podStartE2EDuration="2.113387669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:05:08.11264629 +0000 UTC m=+1067.003503312" watchObservedRunningTime="2026-11-04 05:05:08.113387669 +0000 UTC m=+1067.004244690" Nov 04 05:05:08.532199 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:08.532156 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:08.669410 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:08.669323 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:08.710898 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:08.710856 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:08.711284 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:08.711256 4581 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 04 05:05:08.711445 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:08.711422 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" podUID="b652ed62-2de7-4888-91e1-686213b028f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 04 05:05:09.082693 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:09.082654 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:09.228470 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:09.228390 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" podUID="de6539ee-da6d-40c5-917b-1ecdee1b133c" containerName="main" containerID="cri-o://27532f458c59a2d88879ad2b44a4342c26975abc946bb60285108cf9ca3d1ca2" gracePeriod=45 Nov 04 05:05:09.299293 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:09.299221 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" podUID="081b7b2b-8896-4024-bfaf-4dd8afdd893f" containerName="main" containerID="cri-o://5a44e57c33b97b60452e122d0b42bea592ddc54a7b29096b945307f594054b79" gracePeriod=45 Nov 04 05:05:09.569634 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:09.569328 4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" podUID="de6539ee-da6d-40c5-917b-1ecdee1b133c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": dial tcp 10.134.0.59:8000: connect: connection refused" Nov 04 05:05:09.741276 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:09.739268 4581 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 04 05:05:09.741276 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:09.739349 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" podUID="b652ed62-2de7-4888-91e1-686213b028f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 04 05:05:09.946584 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:09.946393 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:05:10.024763 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.024678 4581 generic.go:356] "Generic (PLEG): container finished" podID="de6539ee-da6d-40c5-917b-1ecdee1b133c" containerID="27532f458c59a2d88879ad2b44a4342c26975abc946bb60285108cf9ca3d1ca2" exitCode=0 Nov 04 05:05:10.024919 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.024805 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" event={"ID":"de6539ee-da6d-40c5-917b-1ecdee1b133c","Type":"ContainerDied","Data":"27532f458c59a2d88879ad2b44a4342c26975abc946bb60285108cf9ca3d1ca2"} Nov 04 05:05:10.024919 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.024842 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" event={"ID":"de6539ee-da6d-40c5-917b-1ecdee1b133c","Type":"ContainerDied","Data":"78e8dda970acc59f81656e5c529ecadf046eb5ae7529c1d6ecf187cb654a4b15"} Nov 04 05:05:10.024919 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.024862 4581 scope.go:122] "RemoveContainer" containerID="27532f458c59a2d88879ad2b44a4342c26975abc946bb60285108cf9ca3d1ca2" Nov 04 05:05:10.025135 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.025028 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7" Nov 04 05:05:10.027200 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.027177 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:05:10.044066 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.044033 4581 generic.go:356] "Generic (PLEG): container finished" podID="081b7b2b-8896-4024-bfaf-4dd8afdd893f" containerID="5a44e57c33b97b60452e122d0b42bea592ddc54a7b29096b945307f594054b79" exitCode=0 Nov 04 05:05:10.045425 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.044412 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" event={"ID":"081b7b2b-8896-4024-bfaf-4dd8afdd893f","Type":"ContainerDied","Data":"5a44e57c33b97b60452e122d0b42bea592ddc54a7b29096b945307f594054b79"} Nov 04 05:05:10.045622 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.045604 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" event={"ID":"081b7b2b-8896-4024-bfaf-4dd8afdd893f","Type":"ContainerDied","Data":"8ea70349cd2e97a06ea17e4d4ea6104d1d4717660282cf8480819f236113b555"} Nov 04 05:05:10.131926 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.131892 4581 scope.go:122] "RemoveContainer" containerID="27532f458c59a2d88879ad2b44a4342c26975abc946bb60285108cf9ca3d1ca2" Nov 04 05:05:10.132699 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:05:10.132671 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27532f458c59a2d88879ad2b44a4342c26975abc946bb60285108cf9ca3d1ca2\": container with ID starting with 27532f458c59a2d88879ad2b44a4342c26975abc946bb60285108cf9ca3d1ca2 not found: ID does not exist" containerID="27532f458c59a2d88879ad2b44a4342c26975abc946bb60285108cf9ca3d1ca2" Nov 04 05:05:10.132878 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.132852 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27532f458c59a2d88879ad2b44a4342c26975abc946bb60285108cf9ca3d1ca2"} err="failed to get container status \"27532f458c59a2d88879ad2b44a4342c26975abc946bb60285108cf9ca3d1ca2\": rpc error: code = NotFound desc = could not find container \"27532f458c59a2d88879ad2b44a4342c26975abc946bb60285108cf9ca3d1ca2\": container with ID starting with 27532f458c59a2d88879ad2b44a4342c26975abc946bb60285108cf9ca3d1ca2 not found: ID does not exist" Nov 04 05:05:10.132979 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.132967 4581 scope.go:122] "RemoveContainer" containerID="5a44e57c33b97b60452e122d0b42bea592ddc54a7b29096b945307f594054b79" Nov 04 05:05:10.137217 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.137188 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/081b7b2b-8896-4024-bfaf-4dd8afdd893f-kube-api-access-lvnpc\" (UniqueName: \"kubernetes.io/projected/081b7b2b-8896-4024-bfaf-4dd8afdd893f-kube-api-access-lvnpc\") pod \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\" (UID: \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\") " Nov 04 05:05:10.137342 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.137249 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-home\" (UniqueName: \"kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-home\") pod \"de6539ee-da6d-40c5-917b-1ecdee1b133c\" (UID: \"de6539ee-da6d-40c5-917b-1ecdee1b133c\") " Nov 04 05:05:10.137342 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.137303 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de6539ee-da6d-40c5-917b-1ecdee1b133c-kube-api-access-kg9pt\" (UniqueName: \"kubernetes.io/projected/de6539ee-da6d-40c5-917b-1ecdee1b133c-kube-api-access-kg9pt\") pod \"de6539ee-da6d-40c5-917b-1ecdee1b133c\" (UID: \"de6539ee-da6d-40c5-917b-1ecdee1b133c\") " Nov 04 05:05:10.137465 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.137353 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-home\" (UniqueName: \"kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-home\") pod \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\" (UID: \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\") " Nov 04 05:05:10.137465 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.137439 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-model-cache\") pod \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\" (UID: \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\") " Nov 04 05:05:10.137584 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.137506 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/081b7b2b-8896-4024-bfaf-4dd8afdd893f-tls-certs\" (UniqueName: \"kubernetes.io/secret/081b7b2b-8896-4024-bfaf-4dd8afdd893f-tls-certs\") pod \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\" (UID: \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\") " Nov 04 05:05:10.137584 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.137542 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-tmp-dir\") pod \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\" (UID: \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\") " Nov 04 05:05:10.137695 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.137582 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-dshm\") pod \"de6539ee-da6d-40c5-917b-1ecdee1b133c\" (UID: \"de6539ee-da6d-40c5-917b-1ecdee1b133c\") " Nov 04 05:05:10.137695 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.137623 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-dshm\") pod \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\" (UID: \"081b7b2b-8896-4024-bfaf-4dd8afdd893f\") " Nov 04 05:05:10.137695 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.137683 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de6539ee-da6d-40c5-917b-1ecdee1b133c-tls-certs\" (UniqueName: \"kubernetes.io/secret/de6539ee-da6d-40c5-917b-1ecdee1b133c-tls-certs\") pod \"de6539ee-da6d-40c5-917b-1ecdee1b133c\" (UID: \"de6539ee-da6d-40c5-917b-1ecdee1b133c\") " Nov 04 05:05:10.137872 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.137720 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-tmp-dir\") pod \"de6539ee-da6d-40c5-917b-1ecdee1b133c\" (UID: \"de6539ee-da6d-40c5-917b-1ecdee1b133c\") " Nov 04 05:05:10.137872 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.137781 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-model-cache\") pod \"de6539ee-da6d-40c5-917b-1ecdee1b133c\" (UID: \"de6539ee-da6d-40c5-917b-1ecdee1b133c\") " Nov 04 05:05:10.143922 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.141657 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-tmp-dir" pod "081b7b2b-8896-4024-bfaf-4dd8afdd893f" (UID: "081b7b2b-8896-4024-bfaf-4dd8afdd893f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:10.143922 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.141853 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-model-cache" pod "de6539ee-da6d-40c5-917b-1ecdee1b133c" (UID: "de6539ee-da6d-40c5-917b-1ecdee1b133c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:10.143922 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.142192 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-home" pod "081b7b2b-8896-4024-bfaf-4dd8afdd893f" (UID: "081b7b2b-8896-4024-bfaf-4dd8afdd893f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:10.143922 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.142485 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-home" pod "de6539ee-da6d-40c5-917b-1ecdee1b133c" (UID: "de6539ee-da6d-40c5-917b-1ecdee1b133c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:10.143922 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.143487 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-model-cache" pod "081b7b2b-8896-4024-bfaf-4dd8afdd893f" (UID: "081b7b2b-8896-4024-bfaf-4dd8afdd893f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:10.143922 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.143834 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-tmp-dir" pod "de6539ee-da6d-40c5-917b-1ecdee1b133c" (UID: "de6539ee-da6d-40c5-917b-1ecdee1b133c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:10.146563 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.146529 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/081b7b2b-8896-4024-bfaf-4dd8afdd893f-tls-certs" pod "081b7b2b-8896-4024-bfaf-4dd8afdd893f" (UID: "081b7b2b-8896-4024-bfaf-4dd8afdd893f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:05:10.147036 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.146964 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de6539ee-da6d-40c5-917b-1ecdee1b133c-kube-api-access-kg9pt" pod "de6539ee-da6d-40c5-917b-1ecdee1b133c" (UID: "de6539ee-da6d-40c5-917b-1ecdee1b133c"). InnerVolumeSpecName "kube-api-access-kg9pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:10.147308 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.147280 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/081b7b2b-8896-4024-bfaf-4dd8afdd893f-kube-api-access-lvnpc" pod "081b7b2b-8896-4024-bfaf-4dd8afdd893f" (UID: "081b7b2b-8896-4024-bfaf-4dd8afdd893f"). InnerVolumeSpecName "kube-api-access-lvnpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:10.147904 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.147877 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-dshm" pod "081b7b2b-8896-4024-bfaf-4dd8afdd893f" (UID: "081b7b2b-8896-4024-bfaf-4dd8afdd893f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:10.148079 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.148060 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-dshm" pod "de6539ee-da6d-40c5-917b-1ecdee1b133c" (UID: "de6539ee-da6d-40c5-917b-1ecdee1b133c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:10.148665 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.148647 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de6539ee-da6d-40c5-917b-1ecdee1b133c-tls-certs" pod "de6539ee-da6d-40c5-917b-1ecdee1b133c" (UID: "de6539ee-da6d-40c5-917b-1ecdee1b133c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:05:10.228638 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.226385 4581 scope.go:122] "RemoveContainer" containerID="5a44e57c33b97b60452e122d0b42bea592ddc54a7b29096b945307f594054b79" Nov 04 05:05:10.228638 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:05:10.228504 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a44e57c33b97b60452e122d0b42bea592ddc54a7b29096b945307f594054b79\": container with ID starting with 5a44e57c33b97b60452e122d0b42bea592ddc54a7b29096b945307f594054b79 not found: ID does not exist" containerID="5a44e57c33b97b60452e122d0b42bea592ddc54a7b29096b945307f594054b79" Nov 04 05:05:10.228638 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.228548 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a44e57c33b97b60452e122d0b42bea592ddc54a7b29096b945307f594054b79"} err="failed to get container status \"5a44e57c33b97b60452e122d0b42bea592ddc54a7b29096b945307f594054b79\": rpc error: code = NotFound desc = could not find container \"5a44e57c33b97b60452e122d0b42bea592ddc54a7b29096b945307f594054b79\": container with ID starting with 5a44e57c33b97b60452e122d0b42bea592ddc54a7b29096b945307f594054b79 not found: ID does not exist" Nov 04 05:05:10.248126 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.243029 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvnpc\" (UniqueName: \"kubernetes.io/projected/081b7b2b-8896-4024-bfaf-4dd8afdd893f-kube-api-access-lvnpc\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:10.248126 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.243067 4581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-home\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:10.248126 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.243085 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg9pt\" (UniqueName: \"kubernetes.io/projected/de6539ee-da6d-40c5-917b-1ecdee1b133c-kube-api-access-kg9pt\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:10.248126 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.243100 4581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-home\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:10.248126 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.243116 4581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-model-cache\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:10.248126 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.243129 4581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/081b7b2b-8896-4024-bfaf-4dd8afdd893f-tls-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:10.248126 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.243144 4581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-tmp-dir\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:10.248126 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.243156 4581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-dshm\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:10.248126 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.243168 4581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/081b7b2b-8896-4024-bfaf-4dd8afdd893f-dshm\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:10.248126 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.243180 4581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de6539ee-da6d-40c5-917b-1ecdee1b133c-tls-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:10.248126 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.243192 4581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-tmp-dir\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:10.248126 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.243205 4581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de6539ee-da6d-40c5-917b-1ecdee1b133c-model-cache\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:10.495349 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.495305 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7"] Nov 04 05:05:10.497159 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.497122 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3eb31c/distinct-2-w4-3eb31c-kserve-7bd44b45-k8rb7"] Nov 04 05:05:10.708822 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.708782 4581 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 04 05:05:10.709072 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.708842 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" podUID="b652ed62-2de7-4888-91e1-686213b028f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 04 05:05:10.778093 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:10.778057 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:11.077071 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:11.071264 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd" Nov 04 05:05:11.120383 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:11.120349 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:11.231479 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:11.231437 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de6539ee-da6d-40c5-917b-1ecdee1b133c" path="/var/lib/kubelet/pods/de6539ee-da6d-40c5-917b-1ecdee1b133c/volumes" Nov 04 05:05:11.241604 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:11.241570 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd"] Nov 04 05:05:11.291586 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:11.291548 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3eb31c/distinct-w4-3eb31c-kserve-89ff97c4b-845sd"] Nov 04 05:05:11.709283 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:11.709251 4581 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 04 05:05:11.709446 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:11.709305 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" podUID="b652ed62-2de7-4888-91e1-686213b028f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 04 05:05:12.347736 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:12.347697 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:12.709566 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:12.709535 4581 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 04 05:05:12.709757 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:12.709595 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" podUID="b652ed62-2de7-4888-91e1-686213b028f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 04 05:05:13.230594 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:13.230558 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="081b7b2b-8896-4024-bfaf-4dd8afdd893f" path="/var/lib/kubelet/pods/081b7b2b-8896-4024-bfaf-4dd8afdd893f/volumes" Nov 04 05:05:13.708823 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:13.708781 4581 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 04 05:05:13.709218 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:13.708843 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" podUID="b652ed62-2de7-4888-91e1-686213b028f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 04 05:05:14.215861 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:14.215821 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:14.713027 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:14.713000 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:14.713493 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:14.713475 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:14.714252 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:14.714227 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:05:18.008034 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:05:18.003634 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 05:05:18.045385 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:18.045350 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:18.177439 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:18.177404 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:19.167388 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:19.167351 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:19.738916 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:19.738874 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:20.298299 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:20.298268 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:20.310039 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:20.310003 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2"] Nov 04 05:05:20.310302 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:20.310276 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" podUID="71ef48c5-d75d-4a0c-a0d8-43b1d2aec689" containerName="istio-proxy" containerID="cri-o://b14948f09af000f39fb24911053dd5000dbf031fc5a7edc74aec637bb47598df" gracePeriod=30 Nov 04 05:05:22.856192 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:22.856151 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:22.978673 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:22.978635 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:23.849462 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:23.849421 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:23.996376 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:23.996344 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:24.937276 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:24.937234 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln"] Nov 04 05:05:24.937547 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:24.937520 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" podUID="0e8292c4-8027-4026-ba22-efc5004ff544" containerName="maas-api" containerID="cri-o://de5678a5ee6d14e44d88ded2f59234c5fa0e569b134bdd130aa31d5df298408c" gracePeriod=30 Nov 04 05:05:25.022884 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.022847 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:25.574111 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.574085 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:05:25.675313 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.675285 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-envoy\") pod \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " Nov 04 05:05:25.675484 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.675403 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-token\" (UniqueName: \"kubernetes.io/projected/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-token\") pod \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " Nov 04 05:05:25.675484 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.675429 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-workload-certs\") pod \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " Nov 04 05:05:25.675484 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.675449 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-workload-socket\") pod \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " Nov 04 05:05:25.675610 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.675481 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-credential-socket\") pod \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " Nov 04 05:05:25.675610 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.675532 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istiod-ca-cert\") pod \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " Nov 04 05:05:25.675610 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.675604 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-podinfo\") pod \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " Nov 04 05:05:25.675763 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.675638 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-data\") pod \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " Nov 04 05:05:25.675763 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.675674 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-kube-api-access-w4xff\" (UniqueName: \"kubernetes.io/projected/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-kube-api-access-w4xff\") pod \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\" (UID: \"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689\") " Nov 04 05:05:25.675763 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.675726 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-workload-certs" pod "71ef48c5-d75d-4a0c-a0d8-43b1d2aec689" (UID: "71ef48c5-d75d-4a0c-a0d8-43b1d2aec689"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:25.675928 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.675761 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-workload-socket" pod "71ef48c5-d75d-4a0c-a0d8-43b1d2aec689" (UID: "71ef48c5-d75d-4a0c-a0d8-43b1d2aec689"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:25.676029 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.675877 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-credential-socket" pod "71ef48c5-d75d-4a0c-a0d8-43b1d2aec689" (UID: "71ef48c5-d75d-4a0c-a0d8-43b1d2aec689"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:25.676029 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.675946 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-data" pod "71ef48c5-d75d-4a0c-a0d8-43b1d2aec689" (UID: "71ef48c5-d75d-4a0c-a0d8-43b1d2aec689"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:25.676237 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.676199 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istiod-ca-cert" pod "71ef48c5-d75d-4a0c-a0d8-43b1d2aec689" (UID: "71ef48c5-d75d-4a0c-a0d8-43b1d2aec689"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:05:25.676367 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.676246 4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-workload-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:25.676367 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.676266 4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-workload-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:25.676367 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.676279 4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-credential-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:25.676367 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.676293 4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-data\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:25.677762 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.677739 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-envoy" pod "71ef48c5-d75d-4a0c-a0d8-43b1d2aec689" (UID: "71ef48c5-d75d-4a0c-a0d8-43b1d2aec689"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:25.677762 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.677745 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-token" pod "71ef48c5-d75d-4a0c-a0d8-43b1d2aec689" (UID: "71ef48c5-d75d-4a0c-a0d8-43b1d2aec689"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:25.678164 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.678139 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-podinfo" pod "71ef48c5-d75d-4a0c-a0d8-43b1d2aec689" (UID: "71ef48c5-d75d-4a0c-a0d8-43b1d2aec689"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:05:25.678403 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.678386 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-kube-api-access-w4xff" pod "71ef48c5-d75d-4a0c-a0d8-43b1d2aec689" (UID: "71ef48c5-d75d-4a0c-a0d8-43b1d2aec689"). InnerVolumeSpecName "kube-api-access-w4xff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:25.777232 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.777192 4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istiod-ca-cert\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:25.777232 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.777231 4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-podinfo\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:25.777232 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.777247 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4xff\" (UniqueName: \"kubernetes.io/projected/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-kube-api-access-w4xff\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:25.777497 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.777263 4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-envoy\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:25.777497 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:25.777276 4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689-istio-token\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:26.097370 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.097338 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:26.183439 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.183417 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" Nov 04 05:05:26.202057 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.201972 4581 generic.go:356] "Generic (PLEG): container finished" podID="0e8292c4-8027-4026-ba22-efc5004ff544" containerID="de5678a5ee6d14e44d88ded2f59234c5fa0e569b134bdd130aa31d5df298408c" exitCode=0 Nov 04 05:05:26.202057 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.202051 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" Nov 04 05:05:26.202282 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.202025 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" event={"ID":"0e8292c4-8027-4026-ba22-efc5004ff544","Type":"ContainerDied","Data":"de5678a5ee6d14e44d88ded2f59234c5fa0e569b134bdd130aa31d5df298408c"} Nov 04 05:05:26.202282 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.202169 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln" event={"ID":"0e8292c4-8027-4026-ba22-efc5004ff544","Type":"ContainerDied","Data":"9ec6a89c6a972a206f6d18cd1e254cfbf01462df7596c306f73dc461257a9d54"} Nov 04 05:05:26.202282 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.202200 4581 scope.go:122] "RemoveContainer" containerID="de5678a5ee6d14e44d88ded2f59234c5fa0e569b134bdd130aa31d5df298408c" Nov 04 05:05:26.203503 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.203483 4581 generic.go:356] "Generic (PLEG): container finished" podID="71ef48c5-d75d-4a0c-a0d8-43b1d2aec689" containerID="b14948f09af000f39fb24911053dd5000dbf031fc5a7edc74aec637bb47598df" exitCode=0 Nov 04 05:05:26.203629 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.203515 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" event={"ID":"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689","Type":"ContainerDied","Data":"b14948f09af000f39fb24911053dd5000dbf031fc5a7edc74aec637bb47598df"} Nov 04 05:05:26.203629 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.203541 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" Nov 04 05:05:26.203629 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.203566 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2" event={"ID":"71ef48c5-d75d-4a0c-a0d8-43b1d2aec689","Type":"ContainerDied","Data":"32d2a5b6bae458b6e6b568a2c4e3d327ecbc8192e185b62b7f6da3720aab2a9a"} Nov 04 05:05:26.212831 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.212815 4581 scope.go:122] "RemoveContainer" containerID="de5678a5ee6d14e44d88ded2f59234c5fa0e569b134bdd130aa31d5df298408c" Nov 04 05:05:26.213144 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:05:26.213125 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5678a5ee6d14e44d88ded2f59234c5fa0e569b134bdd130aa31d5df298408c\": container with ID starting with de5678a5ee6d14e44d88ded2f59234c5fa0e569b134bdd130aa31d5df298408c not found: ID does not exist" containerID="de5678a5ee6d14e44d88ded2f59234c5fa0e569b134bdd130aa31d5df298408c" Nov 04 05:05:26.213232 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.213152 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5678a5ee6d14e44d88ded2f59234c5fa0e569b134bdd130aa31d5df298408c"} err="failed to get container status \"de5678a5ee6d14e44d88ded2f59234c5fa0e569b134bdd130aa31d5df298408c\": rpc error: code = NotFound desc = could not find container \"de5678a5ee6d14e44d88ded2f59234c5fa0e569b134bdd130aa31d5df298408c\": container with ID starting with de5678a5ee6d14e44d88ded2f59234c5fa0e569b134bdd130aa31d5df298408c not found: ID does not exist" Nov 04 05:05:26.213232 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.213172 4581 scope.go:122] "RemoveContainer" containerID="b14948f09af000f39fb24911053dd5000dbf031fc5a7edc74aec637bb47598df" Nov 04 05:05:26.222748 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.222731 4581 scope.go:122] "RemoveContainer" containerID="b14948f09af000f39fb24911053dd5000dbf031fc5a7edc74aec637bb47598df" Nov 04 05:05:26.223042 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:05:26.223023 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b14948f09af000f39fb24911053dd5000dbf031fc5a7edc74aec637bb47598df\": container with ID starting with b14948f09af000f39fb24911053dd5000dbf031fc5a7edc74aec637bb47598df not found: ID does not exist" containerID="b14948f09af000f39fb24911053dd5000dbf031fc5a7edc74aec637bb47598df" Nov 04 05:05:26.223111 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.223054 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b14948f09af000f39fb24911053dd5000dbf031fc5a7edc74aec637bb47598df"} err="failed to get container status \"b14948f09af000f39fb24911053dd5000dbf031fc5a7edc74aec637bb47598df\": rpc error: code = NotFound desc = could not find container \"b14948f09af000f39fb24911053dd5000dbf031fc5a7edc74aec637bb47598df\": container with ID starting with b14948f09af000f39fb24911053dd5000dbf031fc5a7edc74aec637bb47598df not found: ID does not exist" Nov 04 05:05:26.283846 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.283770 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e8292c4-8027-4026-ba22-efc5004ff544-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e8292c4-8027-4026-ba22-efc5004ff544-metrics-certs\") pod \"0e8292c4-8027-4026-ba22-efc5004ff544\" (UID: \"0e8292c4-8027-4026-ba22-efc5004ff544\") " Nov 04 05:05:26.283846 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.283832 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e8292c4-8027-4026-ba22-efc5004ff544-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e8292c4-8027-4026-ba22-efc5004ff544-openshift-service-ca\") pod \"0e8292c4-8027-4026-ba22-efc5004ff544\" (UID: \"0e8292c4-8027-4026-ba22-efc5004ff544\") " Nov 04 05:05:26.284116 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.283863 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e8292c4-8027-4026-ba22-efc5004ff544-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e8292c4-8027-4026-ba22-efc5004ff544-maas-api-tls\") pod \"0e8292c4-8027-4026-ba22-efc5004ff544\" (UID: \"0e8292c4-8027-4026-ba22-efc5004ff544\") " Nov 04 05:05:26.284116 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.283964 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e8292c4-8027-4026-ba22-efc5004ff544-kube-api-access-t98nf\" (UniqueName: \"kubernetes.io/projected/0e8292c4-8027-4026-ba22-efc5004ff544-kube-api-access-t98nf\") pod \"0e8292c4-8027-4026-ba22-efc5004ff544\" (UID: \"0e8292c4-8027-4026-ba22-efc5004ff544\") " Nov 04 05:05:26.284289 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.284263 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e8292c4-8027-4026-ba22-efc5004ff544-openshift-service-ca" pod "0e8292c4-8027-4026-ba22-efc5004ff544" (UID: "0e8292c4-8027-4026-ba22-efc5004ff544"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:05:26.286237 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.286213 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e8292c4-8027-4026-ba22-efc5004ff544-maas-api-tls" pod "0e8292c4-8027-4026-ba22-efc5004ff544" (UID: "0e8292c4-8027-4026-ba22-efc5004ff544"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:05:26.286335 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.286243 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8292c4-8027-4026-ba22-efc5004ff544-kube-api-access-t98nf" pod "0e8292c4-8027-4026-ba22-efc5004ff544" (UID: "0e8292c4-8027-4026-ba22-efc5004ff544"). InnerVolumeSpecName "kube-api-access-t98nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:26.286335 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.286253 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e8292c4-8027-4026-ba22-efc5004ff544-metrics-certs" pod "0e8292c4-8027-4026-ba22-efc5004ff544" (UID: "0e8292c4-8027-4026-ba22-efc5004ff544"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:05:26.385611 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.385580 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t98nf\" (UniqueName: \"kubernetes.io/projected/0e8292c4-8027-4026-ba22-efc5004ff544-kube-api-access-t98nf\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:26.385767 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.385679 4581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e8292c4-8027-4026-ba22-efc5004ff544-metrics-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:26.385767 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.385689 4581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e8292c4-8027-4026-ba22-efc5004ff544-openshift-service-ca\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:26.385767 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.385698 4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e8292c4-8027-4026-ba22-efc5004ff544-maas-api-tls\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:26.445225 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.445194 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:26.465299 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.465266 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2"] Nov 04 05:05:26.470460 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.470432 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2"] Nov 04 05:05:26.573240 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.573147 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:26.626132 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.626095 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln"] Nov 04 05:05:26.629126 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.629104 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3eb31c-85489b5996-b8rln"] Nov 04 05:05:26.807577 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:26.807545 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:27.156920 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:27.156889 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:27.233043 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:27.233010 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e8292c4-8027-4026-ba22-efc5004ff544" path="/var/lib/kubelet/pods/0e8292c4-8027-4026-ba22-efc5004ff544/volumes" Nov 04 05:05:27.233571 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:27.233549 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71ef48c5-d75d-4a0c-a0d8-43b1d2aec689" path="/var/lib/kubelet/pods/71ef48c5-d75d-4a0c-a0d8-43b1d2aec689/volumes" Nov 04 05:05:27.275952 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:27.275917 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:27.949954 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:27.949917 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:28.485405 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:28.485376 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:28.502900 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:28.502871 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n"] Nov 04 05:05:28.503133 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:28.503110 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" podUID="45926cc8-a8ed-40b8-a5f0-db1336dcc102" containerName="istio-proxy" containerID="cri-o://ca55c3fe86220011622c736c4bf21d215f6f3db904b7e564a31f26b0af8b133c" gracePeriod=30 Nov 04 05:05:29.918857 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:29.918823 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:31.522223 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:31.522189 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:31.643621 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:31.643583 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:32.073361 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:32.073326 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:32.202174 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:32.202144 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:33.132998 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.132962 4581 patch_prober.go:29] interesting pod/e2e-worker-w2-370078-openshift-default-584c45775-hp82n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 05:05:33.133370 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.133047 4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" podUID="45926cc8-a8ed-40b8-a5f0-db1336dcc102" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 05:05:33.766437 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.766372 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 05:05:33.871033 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.871003 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-workload-socket\") pod \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " Nov 04 05:05:33.871208 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.871060 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-token\" (UniqueName: \"kubernetes.io/projected/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-token\") pod \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " Nov 04 05:05:33.871208 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.871091 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-credential-socket\") pod \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " Nov 04 05:05:33.871208 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.871129 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-envoy\") pod \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " Nov 04 05:05:33.871384 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.871231 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-podinfo\") pod \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " Nov 04 05:05:33.871384 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.871292 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45926cc8-a8ed-40b8-a5f0-db1336dcc102-kube-api-access-dlw65\" (UniqueName: \"kubernetes.io/projected/45926cc8-a8ed-40b8-a5f0-db1336dcc102-kube-api-access-dlw65\") pod \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " Nov 04 05:05:33.871496 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.871224 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-workload-socket" pod "45926cc8-a8ed-40b8-a5f0-db1336dcc102" (UID: "45926cc8-a8ed-40b8-a5f0-db1336dcc102"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:33.871558 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.871504 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-data\") pod \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " Nov 04 05:05:33.871558 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.871540 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istiod-ca-cert\") pod \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " Nov 04 05:05:33.871660 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.871568 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-workload-certs\") pod \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\" (UID: \"45926cc8-a8ed-40b8-a5f0-db1336dcc102\") " Nov 04 05:05:33.872127 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.872104 4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-workload-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:33.872479 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.872457 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-workload-certs" pod "45926cc8-a8ed-40b8-a5f0-db1336dcc102" (UID: "45926cc8-a8ed-40b8-a5f0-db1336dcc102"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:33.872641 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.872464 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istiod-ca-cert" pod "45926cc8-a8ed-40b8-a5f0-db1336dcc102" (UID: "45926cc8-a8ed-40b8-a5f0-db1336dcc102"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:05:33.872806 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.872540 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-data" pod "45926cc8-a8ed-40b8-a5f0-db1336dcc102" (UID: "45926cc8-a8ed-40b8-a5f0-db1336dcc102"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:33.872931 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.872855 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-credential-socket" pod "45926cc8-a8ed-40b8-a5f0-db1336dcc102" (UID: "45926cc8-a8ed-40b8-a5f0-db1336dcc102"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:33.874477 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.874445 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-podinfo" pod "45926cc8-a8ed-40b8-a5f0-db1336dcc102" (UID: "45926cc8-a8ed-40b8-a5f0-db1336dcc102"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:05:33.874789 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.874746 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-token" pod "45926cc8-a8ed-40b8-a5f0-db1336dcc102" (UID: "45926cc8-a8ed-40b8-a5f0-db1336dcc102"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:33.875132 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.875105 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-envoy" pod "45926cc8-a8ed-40b8-a5f0-db1336dcc102" (UID: "45926cc8-a8ed-40b8-a5f0-db1336dcc102"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:33.875241 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.875122 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45926cc8-a8ed-40b8-a5f0-db1336dcc102-kube-api-access-dlw65" pod "45926cc8-a8ed-40b8-a5f0-db1336dcc102" (UID: "45926cc8-a8ed-40b8-a5f0-db1336dcc102"). InnerVolumeSpecName "kube-api-access-dlw65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:33.973267 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.973232 4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-data\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:33.973267 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.973262 4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istiod-ca-cert\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:33.973267 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.973272 4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-workload-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:33.973492 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.973280 4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-token\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:33.973492 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.973289 4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-credential-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:33.973492 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.973299 4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-envoy\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:33.973492 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.973307 4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45926cc8-a8ed-40b8-a5f0-db1336dcc102-istio-podinfo\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:33.973492 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:33.973315 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlw65\" (UniqueName: \"kubernetes.io/projected/45926cc8-a8ed-40b8-a5f0-db1336dcc102-kube-api-access-dlw65\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:34.094555 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:34.094456 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:34.252744 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:34.252708 4581 generic.go:356] "Generic (PLEG): container finished" podID="45926cc8-a8ed-40b8-a5f0-db1336dcc102" containerID="ca55c3fe86220011622c736c4bf21d215f6f3db904b7e564a31f26b0af8b133c" exitCode=0 Nov 04 05:05:34.253175 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:34.252768 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" Nov 04 05:05:34.253175 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:34.252810 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" event={"ID":"45926cc8-a8ed-40b8-a5f0-db1336dcc102","Type":"ContainerDied","Data":"ca55c3fe86220011622c736c4bf21d215f6f3db904b7e564a31f26b0af8b133c"} Nov 04 05:05:34.253175 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:34.252866 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n" event={"ID":"45926cc8-a8ed-40b8-a5f0-db1336dcc102","Type":"ContainerDied","Data":"72a1fbc84cf392a42b1497d6c5703454bb506e37242ccf5f29c258ce5e82b575"} Nov 04 05:05:34.253175 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:34.252890 4581 scope.go:122] "RemoveContainer" containerID="ca55c3fe86220011622c736c4bf21d215f6f3db904b7e564a31f26b0af8b133c" Nov 04 05:05:34.262914 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:34.262893 4581 scope.go:122] "RemoveContainer" containerID="95c9490826687caa0924e3da386f02b17adf3f868cfa41e36fe1a01f0c106350" Nov 04 05:05:34.271824 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:34.271810 4581 scope.go:122] "RemoveContainer" containerID="ca55c3fe86220011622c736c4bf21d215f6f3db904b7e564a31f26b0af8b133c" Nov 04 05:05:34.272119 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:05:34.272100 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca55c3fe86220011622c736c4bf21d215f6f3db904b7e564a31f26b0af8b133c\": container with ID starting with ca55c3fe86220011622c736c4bf21d215f6f3db904b7e564a31f26b0af8b133c not found: ID does not exist" containerID="ca55c3fe86220011622c736c4bf21d215f6f3db904b7e564a31f26b0af8b133c" Nov 04 05:05:34.272172 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:34.272128 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca55c3fe86220011622c736c4bf21d215f6f3db904b7e564a31f26b0af8b133c"} err="failed to get container status \"ca55c3fe86220011622c736c4bf21d215f6f3db904b7e564a31f26b0af8b133c\": rpc error: code = NotFound desc = could not find container \"ca55c3fe86220011622c736c4bf21d215f6f3db904b7e564a31f26b0af8b133c\": container with ID starting with ca55c3fe86220011622c736c4bf21d215f6f3db904b7e564a31f26b0af8b133c not found: ID does not exist" Nov 04 05:05:34.272172 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:34.272144 4581 scope.go:122] "RemoveContainer" containerID="95c9490826687caa0924e3da386f02b17adf3f868cfa41e36fe1a01f0c106350" Nov 04 05:05:34.272392 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:05:34.272372 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c9490826687caa0924e3da386f02b17adf3f868cfa41e36fe1a01f0c106350\": container with ID starting with 95c9490826687caa0924e3da386f02b17adf3f868cfa41e36fe1a01f0c106350 not found: ID does not exist" containerID="95c9490826687caa0924e3da386f02b17adf3f868cfa41e36fe1a01f0c106350" Nov 04 05:05:34.272433 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:34.272400 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c9490826687caa0924e3da386f02b17adf3f868cfa41e36fe1a01f0c106350"} err="failed to get container status \"95c9490826687caa0924e3da386f02b17adf3f868cfa41e36fe1a01f0c106350\": rpc error: code = NotFound desc = could not find container \"95c9490826687caa0924e3da386f02b17adf3f868cfa41e36fe1a01f0c106350\": container with ID starting with 95c9490826687caa0924e3da386f02b17adf3f868cfa41e36fe1a01f0c106350 not found: ID does not exist" Nov 04 05:05:34.420090 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:34.420061 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n"] Nov 04 05:05:34.470602 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:34.470574 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n"] Nov 04 05:05:35.230156 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:35.230115 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45926cc8-a8ed-40b8-a5f0-db1336dcc102" path="/var/lib/kubelet/pods/45926cc8-a8ed-40b8-a5f0-db1336dcc102/volumes" Nov 04 05:05:35.300846 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:35.300810 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:35.858107 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:35.858027 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:35.861583 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:35.861554 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq"] Nov 04 05:05:35.862062 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:35.862033 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" podUID="2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2" containerName="istio-proxy" containerID="cri-o://05f600a8213dcdb5c37fac19e48c3680a01464c59d9205f63bfd30d195dd93cf" gracePeriod=30 Nov 04 05:05:35.866032 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:35.864929 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp"] Nov 04 05:05:35.866032 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:35.865139 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" podUID="f9ac68d4-0a7a-47aa-afeb-803dfc3f362a" containerName="istio-proxy" containerID="cri-o://3afdef2abe8f0af45f9f4c7ef6532d1be73155eca2267b393dfb509ba0cb7832" gracePeriod=30 Nov 04 05:05:35.906804 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:35.906762 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56"] Nov 04 05:05:35.907059 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:35.907020 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" podUID="1ce6a777-2cfb-4cf5-905a-209d8d82f331" containerName="istio-proxy" containerID="cri-o://4a42851e8dafb92187ca246ee09c34582945257d023a7421cde6c7359cb206e2" gracePeriod=30 Nov 04 05:05:37.072143 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.072102 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:37.208855 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.208823 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776"] Nov 04 05:05:37.209377 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.209358 4581 cpu_manager.go:414] "containerMap: removing container" podUID="de6539ee-da6d-40c5-917b-1ecdee1b133c" containerName="main" Nov 04 05:05:37.209488 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.209379 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de6539ee-da6d-40c5-917b-1ecdee1b133c" containerName="main" Nov 04 05:05:37.209488 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.209409 4581 cpu_manager.go:414] "containerMap: removing container" podUID="ad5ae17b-aca8-4988-8ed1-7042c62fc32f" containerName="istio-proxy" Nov 04 05:05:37.209488 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.209417 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad5ae17b-aca8-4988-8ed1-7042c62fc32f" containerName="istio-proxy" Nov 04 05:05:37.209488 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.209431 4581 cpu_manager.go:414] "containerMap: removing container" podUID="0e8292c4-8027-4026-ba22-efc5004ff544" containerName="maas-api" Nov 04 05:05:37.209488 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.209440 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8292c4-8027-4026-ba22-efc5004ff544" containerName="maas-api" Nov 04 05:05:37.209488 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.209455 4581 cpu_manager.go:414] "containerMap: removing container" podUID="081b7b2b-8896-4024-bfaf-4dd8afdd893f" containerName="main" Nov 04 05:05:37.209488 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.209462 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="081b7b2b-8896-4024-bfaf-4dd8afdd893f" containerName="main" Nov 04 05:05:37.209488 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.209489 4581 cpu_manager.go:414] "containerMap: removing container" podUID="45926cc8-a8ed-40b8-a5f0-db1336dcc102" containerName="istio-proxy" Nov 04 05:05:37.209880 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.209496 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45926cc8-a8ed-40b8-a5f0-db1336dcc102" containerName="istio-proxy" Nov 04 05:05:37.209880 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.209522 4581 cpu_manager.go:414] "containerMap: removing container" podUID="71ef48c5-d75d-4a0c-a0d8-43b1d2aec689" containerName="istio-proxy" Nov 04 05:05:37.209880 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.209529 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ef48c5-d75d-4a0c-a0d8-43b1d2aec689" containerName="istio-proxy" Nov 04 05:05:37.209880 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.209538 4581 cpu_manager.go:414] "containerMap: removing container" podUID="45926cc8-a8ed-40b8-a5f0-db1336dcc102" containerName="istio-proxy" Nov 04 05:05:37.209880 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.209545 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45926cc8-a8ed-40b8-a5f0-db1336dcc102" containerName="istio-proxy" Nov 04 05:05:37.209880 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.209649 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e8292c4-8027-4026-ba22-efc5004ff544" containerName="maas-api" Nov 04 05:05:37.209880 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.209661 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="081b7b2b-8896-4024-bfaf-4dd8afdd893f" containerName="main" Nov 04 05:05:37.209880 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.209673 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="45926cc8-a8ed-40b8-a5f0-db1336dcc102" containerName="istio-proxy" Nov 04 05:05:37.209880 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.209682 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad5ae17b-aca8-4988-8ed1-7042c62fc32f" containerName="istio-proxy" Nov 04 05:05:37.209880 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.209693 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ef48c5-d75d-4a0c-a0d8-43b1d2aec689" containerName="istio-proxy" Nov 04 05:05:37.209880 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.209701 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="45926cc8-a8ed-40b8-a5f0-db1336dcc102" containerName="istio-proxy" Nov 04 05:05:37.209880 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.209711 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="de6539ee-da6d-40c5-917b-1ecdee1b133c" containerName="main" Nov 04 05:05:37.214690 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.214670 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.228584 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.228560 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:37.271375 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.271354 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-cedfca-openshift-default-dockercfg-fdtt2\"" Nov 04 05:05:37.324331 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.324262 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776"] Nov 04 05:05:37.412914 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.412883 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-workload-socket\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.413107 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.412922 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-podinfo\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.413107 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.412953 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-envoy\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.413107 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.413042 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-data\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.413107 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.413094 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-token\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.413315 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.413133 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istiod-ca-cert\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.413315 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.413158 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-workload-certs\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.413410 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.413337 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-credential-socket\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.413410 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.413384 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h9hn\" (UniqueName: \"kubernetes.io/projected/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-kube-api-access-4h9hn\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.514613 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.514583 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-token\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.514782 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.514636 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istiod-ca-cert\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.514782 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.514660 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-workload-certs\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.514782 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.514711 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-credential-socket\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.514782 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.514735 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h9hn\" (UniqueName: \"kubernetes.io/projected/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-kube-api-access-4h9hn\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.514782 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.514780 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-workload-socket\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.515088 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.514811 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-podinfo\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.515088 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.514856 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-envoy\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.515088 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.514888 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-data\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.515184 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.515110 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-workload-certs\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.515471 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.515442 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istiod-ca-cert\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.515613 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.515487 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-workload-socket\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.515613 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.515525 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-credential-socket\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.515778 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.515657 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-data\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.517407 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.517379 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-envoy\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.517500 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.517414 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-podinfo\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.571643 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.571620 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-token\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.571844 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.571826 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h9hn\" (UniqueName: \"kubernetes.io/projected/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-kube-api-access-4h9hn\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:37.828388 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:37.828355 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:38.258126 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:38.258103 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776"] Nov 04 05:05:38.260002 ip-10-0-141-132 kubenswrapper[4581]: W1104 05:05:38.259964 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c643f4_9bbd_48f1_a6eb_819bc48c352e.slice/crio-f861085e25e4e094d5774491fff050f3f17c1d6db8b4aaf5539a291c88d2d2a9 WatchSource:0}: Error finding container f861085e25e4e094d5774491fff050f3f17c1d6db8b4aaf5539a291c88d2d2a9: Status 404 returned error can't find the container with id f861085e25e4e094d5774491fff050f3f17c1d6db8b4aaf5539a291c88d2d2a9 Nov 04 05:05:38.262269 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:38.262237 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:05:38.264022 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:38.262868 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:05:38.264022 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:38.263032 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:05:38.281054 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:38.281014 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" event={"ID":"c1c643f4-9bbd-48f1-a6eb-819bc48c352e","Type":"ContainerStarted","Data":"f861085e25e4e094d5774491fff050f3f17c1d6db8b4aaf5539a291c88d2d2a9"} Nov 04 05:05:39.078265 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:39.078225 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr"] Nov 04 05:05:39.209019 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:39.208970 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:39.287972 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:39.287930 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" event={"ID":"c1c643f4-9bbd-48f1-a6eb-819bc48c352e","Type":"ContainerStarted","Data":"d698b2df7c0e3c232d5720a9e54c440c0fa9f919a885411ada3a86b28433a5af"} Nov 04 05:05:39.408717 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:39.408659 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" podStartSLOduration=2.408643766 podStartE2EDuration="2.408643766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:05:39.406179253 +0000 UTC m=+1098.297036272" watchObservedRunningTime="2026-11-04 05:05:39.408643766 +0000 UTC m=+1098.299500786" Nov 04 05:05:39.828779 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:39.828688 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:39.830346 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:39.830320 4581 patch_prober.go:29] interesting pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 04 05:05:39.830476 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:39.830377 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" podUID="c1c643f4-9bbd-48f1-a6eb-819bc48c352e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 04 05:05:40.828996 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:40.828953 4581 patch_prober.go:29] interesting pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 04 05:05:40.829407 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:40.829036 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" podUID="c1c643f4-9bbd-48f1-a6eb-819bc48c352e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 04 05:05:41.197099 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.197055 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 05:05:41.249824 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.249802 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:05:41.284733 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.284701 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-workload-socket\") pod \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " Nov 04 05:05:41.284875 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.284758 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-token\" (UniqueName: \"kubernetes.io/projected/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-token\") pod \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " Nov 04 05:05:41.284875 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.284791 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-workload-certs\") pod \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " Nov 04 05:05:41.284875 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.284820 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istiod-ca-cert\") pod \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " Nov 04 05:05:41.285048 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.284897 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-envoy\") pod \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " Nov 04 05:05:41.285048 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.284965 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-credential-socket\") pod \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " Nov 04 05:05:41.285048 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.285035 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-workload-socket" pod "f9ac68d4-0a7a-47aa-afeb-803dfc3f362a" (UID: "f9ac68d4-0a7a-47aa-afeb-803dfc3f362a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:41.285210 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.285058 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-data\") pod \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " Nov 04 05:05:41.285210 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.285087 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-podinfo\") pod \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " Nov 04 05:05:41.285210 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.285149 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-kube-api-access-8twsr\" (UniqueName: \"kubernetes.io/projected/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-kube-api-access-8twsr\") pod \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\" (UID: \"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a\") " Nov 04 05:05:41.285373 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.285346 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istiod-ca-cert" pod "f9ac68d4-0a7a-47aa-afeb-803dfc3f362a" (UID: "f9ac68d4-0a7a-47aa-afeb-803dfc3f362a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:05:41.285434 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.285355 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-data" pod "f9ac68d4-0a7a-47aa-afeb-803dfc3f362a" (UID: "f9ac68d4-0a7a-47aa-afeb-803dfc3f362a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:41.285572 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.285547 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-workload-certs" pod "f9ac68d4-0a7a-47aa-afeb-803dfc3f362a" (UID: "f9ac68d4-0a7a-47aa-afeb-803dfc3f362a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:41.285665 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.285645 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-credential-socket" pod "f9ac68d4-0a7a-47aa-afeb-803dfc3f362a" (UID: "f9ac68d4-0a7a-47aa-afeb-803dfc3f362a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:41.285935 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.285913 4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-credential-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.286008 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.285937 4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-data\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.286008 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.285952 4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-workload-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.286008 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.285965 4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-workload-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.286008 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.285978 4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istiod-ca-cert\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.287960 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.287936 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-envoy" pod "f9ac68d4-0a7a-47aa-afeb-803dfc3f362a" (UID: "f9ac68d4-0a7a-47aa-afeb-803dfc3f362a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:41.288118 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.287980 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-kube-api-access-8twsr" pod "f9ac68d4-0a7a-47aa-afeb-803dfc3f362a" (UID: "f9ac68d4-0a7a-47aa-afeb-803dfc3f362a"). InnerVolumeSpecName "kube-api-access-8twsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:41.288364 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.288345 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-podinfo" pod "f9ac68d4-0a7a-47aa-afeb-803dfc3f362a" (UID: "f9ac68d4-0a7a-47aa-afeb-803dfc3f362a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:05:41.288627 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.288607 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-token" pod "f9ac68d4-0a7a-47aa-afeb-803dfc3f362a" (UID: "f9ac68d4-0a7a-47aa-afeb-803dfc3f362a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:41.289362 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.289347 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:05:41.299850 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.299828 4581 generic.go:356] "Generic (PLEG): container finished" podID="f9ac68d4-0a7a-47aa-afeb-803dfc3f362a" containerID="3afdef2abe8f0af45f9f4c7ef6532d1be73155eca2267b393dfb509ba0cb7832" exitCode=0 Nov 04 05:05:41.299951 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.299938 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" event={"ID":"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a","Type":"ContainerDied","Data":"3afdef2abe8f0af45f9f4c7ef6532d1be73155eca2267b393dfb509ba0cb7832"} Nov 04 05:05:41.300102 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.299979 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" event={"ID":"f9ac68d4-0a7a-47aa-afeb-803dfc3f362a","Type":"ContainerDied","Data":"39773d1c9849074c83a0f186a2b1393eb492702e675451fc1b57da4c3e95c012"} Nov 04 05:05:41.300102 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.300022 4581 scope.go:122] "RemoveContainer" containerID="3afdef2abe8f0af45f9f4c7ef6532d1be73155eca2267b393dfb509ba0cb7832" Nov 04 05:05:41.300176 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.300117 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp" Nov 04 05:05:41.304827 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.304800 4581 generic.go:356] "Generic (PLEG): container finished" podID="2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2" containerID="05f600a8213dcdb5c37fac19e48c3680a01464c59d9205f63bfd30d195dd93cf" exitCode=0 Nov 04 05:05:41.304950 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.304931 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" event={"ID":"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2","Type":"ContainerDied","Data":"05f600a8213dcdb5c37fac19e48c3680a01464c59d9205f63bfd30d195dd93cf"} Nov 04 05:05:41.305027 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.304978 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" event={"ID":"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2","Type":"ContainerDied","Data":"595fa9326914411f02983b65774a1957077b19dafee548a0860351057aa06018"} Nov 04 05:05:41.305100 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.305087 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq" Nov 04 05:05:41.306625 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.306600 4581 generic.go:356] "Generic (PLEG): container finished" podID="1ce6a777-2cfb-4cf5-905a-209d8d82f331" containerID="4a42851e8dafb92187ca246ee09c34582945257d023a7421cde6c7359cb206e2" exitCode=0 Nov 04 05:05:41.307069 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.307043 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" event={"ID":"1ce6a777-2cfb-4cf5-905a-209d8d82f331","Type":"ContainerDied","Data":"4a42851e8dafb92187ca246ee09c34582945257d023a7421cde6c7359cb206e2"} Nov 04 05:05:41.307136 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.307072 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" Nov 04 05:05:41.307136 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.307106 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56" event={"ID":"1ce6a777-2cfb-4cf5-905a-209d8d82f331","Type":"ContainerDied","Data":"401095da52b231c3c81d29120e1b9cceffc84dda9c0d59db40276af8d05621c5"} Nov 04 05:05:41.315230 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.315213 4581 scope.go:122] "RemoveContainer" containerID="3afdef2abe8f0af45f9f4c7ef6532d1be73155eca2267b393dfb509ba0cb7832" Nov 04 05:05:41.317345 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:05:41.317314 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3afdef2abe8f0af45f9f4c7ef6532d1be73155eca2267b393dfb509ba0cb7832\": container with ID starting with 3afdef2abe8f0af45f9f4c7ef6532d1be73155eca2267b393dfb509ba0cb7832 not found: ID does not exist" containerID="3afdef2abe8f0af45f9f4c7ef6532d1be73155eca2267b393dfb509ba0cb7832" Nov 04 05:05:41.317426 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.317357 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3afdef2abe8f0af45f9f4c7ef6532d1be73155eca2267b393dfb509ba0cb7832"} err="failed to get container status \"3afdef2abe8f0af45f9f4c7ef6532d1be73155eca2267b393dfb509ba0cb7832\": rpc error: code = NotFound desc = could not find container \"3afdef2abe8f0af45f9f4c7ef6532d1be73155eca2267b393dfb509ba0cb7832\": container with ID starting with 3afdef2abe8f0af45f9f4c7ef6532d1be73155eca2267b393dfb509ba0cb7832 not found: ID does not exist" Nov 04 05:05:41.317426 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.317380 4581 scope.go:122] "RemoveContainer" containerID="05f600a8213dcdb5c37fac19e48c3680a01464c59d9205f63bfd30d195dd93cf" Nov 04 05:05:41.329018 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.328502 4581 scope.go:122] "RemoveContainer" containerID="05f600a8213dcdb5c37fac19e48c3680a01464c59d9205f63bfd30d195dd93cf" Nov 04 05:05:41.329018 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:05:41.328805 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05f600a8213dcdb5c37fac19e48c3680a01464c59d9205f63bfd30d195dd93cf\": container with ID starting with 05f600a8213dcdb5c37fac19e48c3680a01464c59d9205f63bfd30d195dd93cf not found: ID does not exist" containerID="05f600a8213dcdb5c37fac19e48c3680a01464c59d9205f63bfd30d195dd93cf" Nov 04 05:05:41.329018 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.328826 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05f600a8213dcdb5c37fac19e48c3680a01464c59d9205f63bfd30d195dd93cf"} err="failed to get container status \"05f600a8213dcdb5c37fac19e48c3680a01464c59d9205f63bfd30d195dd93cf\": rpc error: code = NotFound desc = could not find container \"05f600a8213dcdb5c37fac19e48c3680a01464c59d9205f63bfd30d195dd93cf\": container with ID starting with 05f600a8213dcdb5c37fac19e48c3680a01464c59d9205f63bfd30d195dd93cf not found: ID does not exist" Nov 04 05:05:41.329018 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.328842 4581 scope.go:122] "RemoveContainer" containerID="4a42851e8dafb92187ca246ee09c34582945257d023a7421cde6c7359cb206e2" Nov 04 05:05:41.342148 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.342127 4581 scope.go:122] "RemoveContainer" containerID="4a42851e8dafb92187ca246ee09c34582945257d023a7421cde6c7359cb206e2" Nov 04 05:05:41.342486 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:05:41.342459 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a42851e8dafb92187ca246ee09c34582945257d023a7421cde6c7359cb206e2\": container with ID starting with 4a42851e8dafb92187ca246ee09c34582945257d023a7421cde6c7359cb206e2 not found: ID does not exist" containerID="4a42851e8dafb92187ca246ee09c34582945257d023a7421cde6c7359cb206e2" Nov 04 05:05:41.342570 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.342495 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a42851e8dafb92187ca246ee09c34582945257d023a7421cde6c7359cb206e2"} err="failed to get container status \"4a42851e8dafb92187ca246ee09c34582945257d023a7421cde6c7359cb206e2\": rpc error: code = NotFound desc = could not find container \"4a42851e8dafb92187ca246ee09c34582945257d023a7421cde6c7359cb206e2\": container with ID starting with 4a42851e8dafb92187ca246ee09c34582945257d023a7421cde6c7359cb206e2 not found: ID does not exist" Nov 04 05:05:41.386611 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.386579 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce6a777-2cfb-4cf5-905a-209d8d82f331-kube-api-access-qnfgc\" (UniqueName: \"kubernetes.io/projected/1ce6a777-2cfb-4cf5-905a-209d8d82f331-kube-api-access-qnfgc\") pod \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " Nov 04 05:05:41.386850 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.386619 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-envoy\") pod \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " Nov 04 05:05:41.386850 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.386639 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-podinfo\") pod \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " Nov 04 05:05:41.386850 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.386658 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-workload-certs\") pod \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " Nov 04 05:05:41.386850 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.386674 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istiod-ca-cert\") pod \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " Nov 04 05:05:41.386850 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.386694 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-workload-socket\") pod \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " Nov 04 05:05:41.386850 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.386720 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-envoy\") pod \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " Nov 04 05:05:41.386850 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.386760 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-credential-socket\") pod \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " Nov 04 05:05:41.386850 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.386804 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-workload-socket\") pod \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " Nov 04 05:05:41.386850 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.386826 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-token\" (UniqueName: \"kubernetes.io/projected/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-token\") pod \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " Nov 04 05:05:41.386850 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.386844 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-workload-certs\") pod \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " Nov 04 05:05:41.387185 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.386867 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-token\" (UniqueName: \"kubernetes.io/projected/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-token\") pod \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " Nov 04 05:05:41.387185 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.386891 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istiod-ca-cert\") pod \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " Nov 04 05:05:41.387185 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.386909 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-kube-api-access-lf2gv\" (UniqueName: \"kubernetes.io/projected/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-kube-api-access-lf2gv\") pod \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " Nov 04 05:05:41.387185 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.386925 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-credential-socket\") pod \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " Nov 04 05:05:41.387185 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.386959 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-data\") pod \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " Nov 04 05:05:41.387185 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.387017 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-data\") pod \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\" (UID: \"1ce6a777-2cfb-4cf5-905a-209d8d82f331\") " Nov 04 05:05:41.387185 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.387038 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-podinfo\") pod \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\" (UID: \"2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2\") " Nov 04 05:05:41.387405 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.387393 4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-token\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.387437 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.387407 4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-envoy\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.387437 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.387416 4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-istio-podinfo\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.387437 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.387425 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8twsr\" (UniqueName: \"kubernetes.io/projected/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a-kube-api-access-8twsr\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.387671 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.387639 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-workload-certs" pod "1ce6a777-2cfb-4cf5-905a-209d8d82f331" (UID: "1ce6a777-2cfb-4cf5-905a-209d8d82f331"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:41.387902 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.387881 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-credential-socket" pod "2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2" (UID: "2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:41.387980 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.387948 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-workload-socket" pod "1ce6a777-2cfb-4cf5-905a-209d8d82f331" (UID: "1ce6a777-2cfb-4cf5-905a-209d8d82f331"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:41.388073 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.388050 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-data" pod "1ce6a777-2cfb-4cf5-905a-209d8d82f331" (UID: "1ce6a777-2cfb-4cf5-905a-209d8d82f331"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:41.388134 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.388078 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istiod-ca-cert" pod "2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2" (UID: "2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:05:41.388191 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.388165 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-workload-socket" pod "2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2" (UID: "2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:41.388310 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.388294 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-workload-certs" pod "2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2" (UID: "2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:41.389622 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.388902 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-credential-socket" pod "1ce6a777-2cfb-4cf5-905a-209d8d82f331" (UID: "1ce6a777-2cfb-4cf5-905a-209d8d82f331"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:41.389622 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.389202 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istiod-ca-cert" pod "1ce6a777-2cfb-4cf5-905a-209d8d82f331" (UID: "1ce6a777-2cfb-4cf5-905a-209d8d82f331"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:05:41.389622 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.389489 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-data" pod "2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2" (UID: "2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:41.392697 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.390961 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-token" pod "1ce6a777-2cfb-4cf5-905a-209d8d82f331" (UID: "1ce6a777-2cfb-4cf5-905a-209d8d82f331"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:41.392697 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.391503 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-podinfo" pod "1ce6a777-2cfb-4cf5-905a-209d8d82f331" (UID: "1ce6a777-2cfb-4cf5-905a-209d8d82f331"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:05:41.392697 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.391531 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-envoy" pod "2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2" (UID: "2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:41.392697 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.391576 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-token" pod "2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2" (UID: "2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:41.392697 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.391869 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-podinfo" pod "2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2" (UID: "2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:05:41.393363 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.392882 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-envoy" pod "1ce6a777-2cfb-4cf5-905a-209d8d82f331" (UID: "1ce6a777-2cfb-4cf5-905a-209d8d82f331"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:41.393603 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.393570 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce6a777-2cfb-4cf5-905a-209d8d82f331-kube-api-access-qnfgc" pod "1ce6a777-2cfb-4cf5-905a-209d8d82f331" (UID: "1ce6a777-2cfb-4cf5-905a-209d8d82f331"). InnerVolumeSpecName "kube-api-access-qnfgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:41.395616 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.395561 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-kube-api-access-lf2gv" pod "2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2" (UID: "2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2"). InnerVolumeSpecName "kube-api-access-lf2gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:41.488296 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.488077 4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-workload-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.488296 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.488110 4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-envoy\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.488296 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.488120 4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-credential-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.488296 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.488129 4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-workload-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.488296 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.488138 4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-token\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.488296 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.488147 4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-workload-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.488296 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.488162 4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-token\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.488296 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.488174 4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istiod-ca-cert\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.488296 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.488183 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf2gv\" (UniqueName: \"kubernetes.io/projected/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-kube-api-access-lf2gv\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.488296 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.488191 4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-credential-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.488296 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.488201 4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-data\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.488296 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.488209 4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-data\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.488296 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.488217 4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-podinfo\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.488296 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.488225 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnfgc\" (UniqueName: \"kubernetes.io/projected/1ce6a777-2cfb-4cf5-905a-209d8d82f331-kube-api-access-qnfgc\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.488296 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.488239 4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istio-envoy\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.488296 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.488251 4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce6a777-2cfb-4cf5-905a-209d8d82f331-istio-podinfo\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.488296 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.488259 4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-workload-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.488296 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.488275 4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2-istiod-ca-cert\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.638503 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.638476 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp"] Nov 04 05:05:41.643261 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.643238 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp"] Nov 04 05:05:41.829326 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.829247 4581 patch_prober.go:29] interesting pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 04 05:05:41.829326 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.829304 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" podUID="c1c643f4-9bbd-48f1-a6eb-819bc48c352e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 04 05:05:41.894881 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.894843 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq"] Nov 04 05:05:41.899446 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:41.899421 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq"] Nov 04 05:05:42.012056 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:42.012019 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56"] Nov 04 05:05:42.017419 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:42.017395 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56"] Nov 04 05:05:42.230396 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:42.230365 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:42.828895 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:42.828861 4581 patch_prober.go:29] interesting pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 04 05:05:42.829133 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:42.828922 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" podUID="c1c643f4-9bbd-48f1-a6eb-819bc48c352e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 04 05:05:43.093524 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:43.093448 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:43.229581 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:43.229549 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ce6a777-2cfb-4cf5-905a-209d8d82f331" path="/var/lib/kubelet/pods/1ce6a777-2cfb-4cf5-905a-209d8d82f331/volumes" Nov 04 05:05:43.230017 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:43.230003 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2" path="/var/lib/kubelet/pods/2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2/volumes" Nov 04 05:05:43.230401 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:43.230390 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9ac68d4-0a7a-47aa-afeb-803dfc3f362a" path="/var/lib/kubelet/pods/f9ac68d4-0a7a-47aa-afeb-803dfc3f362a/volumes" Nov 04 05:05:43.829073 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:43.829041 4581 patch_prober.go:29] interesting pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 04 05:05:43.829235 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:43.829092 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" podUID="c1c643f4-9bbd-48f1-a6eb-819bc48c352e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 04 05:05:44.797151 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:44.797114 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:44.828972 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:44.828926 4581 patch_prober.go:29] interesting pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 04 05:05:44.829165 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:44.828998 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" podUID="c1c643f4-9bbd-48f1-a6eb-819bc48c352e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 04 05:05:45.302685 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:45.302640 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:45.832728 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:45.832702 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:45.833159 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:45.832964 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:45.833799 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:45.833781 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:05:46.951352 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:46.951317 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:47.887076 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:05:47.887042 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 05:05:48.421119 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:48.421070 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:51.684844 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:51.684793 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:51.812213 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:51.812171 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:52.634548 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:52.634518 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:52.753385 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:52.753349 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:53.394715 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:53.394673 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl"] Nov 04 05:05:53.444448 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:53.444412 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc"] Nov 04 05:05:53.490771 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:53.490736 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:54.099636 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:54.099591 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" podUID="9414f040-9632-4245-b1c3-b00b229cb75a" containerName="main" containerID="cri-o://18504455d82ab778be4b6847592cac5c263006b128509002f016a5a8a4676d4d" gracePeriod=45 Nov 04 05:05:54.386472 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:54.386432 4581 generic.go:356] "Generic (PLEG): container finished" podID="9414f040-9632-4245-b1c3-b00b229cb75a" containerID="18504455d82ab778be4b6847592cac5c263006b128509002f016a5a8a4676d4d" exitCode=0 Nov 04 05:05:54.386630 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:54.386512 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" event={"ID":"9414f040-9632-4245-b1c3-b00b229cb75a","Type":"ContainerDied","Data":"18504455d82ab778be4b6847592cac5c263006b128509002f016a5a8a4676d4d"} Nov 04 05:05:54.386630 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:54.386546 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" event={"ID":"9414f040-9632-4245-b1c3-b00b229cb75a","Type":"ContainerDied","Data":"1aedc8657e60150f8a3797fb8860ab690f5e2258157006e6148084edca23f21a"} Nov 04 05:05:54.386630 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:54.386563 4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1aedc8657e60150f8a3797fb8860ab690f5e2258157006e6148084edca23f21a" Nov 04 05:05:54.390091 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:54.390063 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:05:54.519364 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:54.519330 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9414f040-9632-4245-b1c3-b00b229cb75a-kube-api-access-psx6w\" (UniqueName: \"kubernetes.io/projected/9414f040-9632-4245-b1c3-b00b229cb75a-kube-api-access-psx6w\") pod \"9414f040-9632-4245-b1c3-b00b229cb75a\" (UID: \"9414f040-9632-4245-b1c3-b00b229cb75a\") " Nov 04 05:05:54.519579 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:54.519395 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-home\" (UniqueName: \"kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-home\") pod \"9414f040-9632-4245-b1c3-b00b229cb75a\" (UID: \"9414f040-9632-4245-b1c3-b00b229cb75a\") " Nov 04 05:05:54.519726 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:54.519675 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-home" pod "9414f040-9632-4245-b1c3-b00b229cb75a" (UID: "9414f040-9632-4245-b1c3-b00b229cb75a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:54.519930 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:54.519903 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-dshm\") pod \"9414f040-9632-4245-b1c3-b00b229cb75a\" (UID: \"9414f040-9632-4245-b1c3-b00b229cb75a\") " Nov 04 05:05:54.520061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:54.519954 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-model-cache\") pod \"9414f040-9632-4245-b1c3-b00b229cb75a\" (UID: \"9414f040-9632-4245-b1c3-b00b229cb75a\") " Nov 04 05:05:54.520124 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:54.520064 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-tmp-dir\") pod \"9414f040-9632-4245-b1c3-b00b229cb75a\" (UID: \"9414f040-9632-4245-b1c3-b00b229cb75a\") " Nov 04 05:05:54.520235 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:54.520208 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-model-cache" pod "9414f040-9632-4245-b1c3-b00b229cb75a" (UID: "9414f040-9632-4245-b1c3-b00b229cb75a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:54.520298 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:54.520214 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9414f040-9632-4245-b1c3-b00b229cb75a-tls-certs\" (UniqueName: \"kubernetes.io/secret/9414f040-9632-4245-b1c3-b00b229cb75a-tls-certs\") pod \"9414f040-9632-4245-b1c3-b00b229cb75a\" (UID: \"9414f040-9632-4245-b1c3-b00b229cb75a\") " Nov 04 05:05:54.520359 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:54.520339 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-tmp-dir" pod "9414f040-9632-4245-b1c3-b00b229cb75a" (UID: "9414f040-9632-4245-b1c3-b00b229cb75a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:54.520863 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:54.520838 4581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-tmp-dir\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:54.520962 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:54.520870 4581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-home\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:54.520962 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:54.520885 4581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-model-cache\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:54.522358 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:54.522330 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-dshm" pod "9414f040-9632-4245-b1c3-b00b229cb75a" (UID: "9414f040-9632-4245-b1c3-b00b229cb75a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:54.522466 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:54.522380 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9414f040-9632-4245-b1c3-b00b229cb75a-kube-api-access-psx6w" pod "9414f040-9632-4245-b1c3-b00b229cb75a" (UID: "9414f040-9632-4245-b1c3-b00b229cb75a"). InnerVolumeSpecName "kube-api-access-psx6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:54.522589 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:54.522573 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9414f040-9632-4245-b1c3-b00b229cb75a-tls-certs" pod "9414f040-9632-4245-b1c3-b00b229cb75a" (UID: "9414f040-9632-4245-b1c3-b00b229cb75a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:05:54.622436 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:54.622351 4581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9414f040-9632-4245-b1c3-b00b229cb75a-dshm\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:54.622436 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:54.622380 4581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9414f040-9632-4245-b1c3-b00b229cb75a-tls-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:54.622436 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:54.622390 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psx6w\" (UniqueName: \"kubernetes.io/projected/9414f040-9632-4245-b1c3-b00b229cb75a-kube-api-access-psx6w\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:55.392449 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:55.392417 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr" Nov 04 05:05:55.558067 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:55.557968 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr"] Nov 04 05:05:55.608198 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:55.608162 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-cedfca/premium-w6-cedfca-kserve-7b859ff974-fj8mr"] Nov 04 05:05:56.326298 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:56.326267 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:57.089979 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:57.089942 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:57.230280 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:57.230249 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9414f040-9632-4245-b1c3-b00b229cb75a" path="/var/lib/kubelet/pods/9414f040-9632-4245-b1c3-b00b229cb75a/volumes" Nov 04 05:05:57.611652 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:57.611616 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:58.018599 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:58.018508 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:05:58.143434 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:05:58.143402 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:00.081188 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:00.081153 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:00.954912 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:00.954879 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:01.938104 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:01.938065 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:02.416191 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:02.416163 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:02.587928 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:02.587897 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:02.986541 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:02.986506 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc"] Nov 04 05:06:03.230414 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:03.230382 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:03.367926 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:03.367879 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:06.401151 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:06.401116 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v"] Nov 04 05:06:06.401592 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:06.401336 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" podUID="00ab6653-dd2e-4dd5-bf65-27428a66e07f" containerName="maas-api" containerID="cri-o://91bbdad1580664b2a96d124b863ad28e1720be1e8a39024517af8bdcf5742f5e" gracePeriod=30 Nov 04 05:06:06.487811 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:06.487780 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:06.668134 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:06.668069 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" Nov 04 05:06:06.849970 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:06.849939 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00ab6653-dd2e-4dd5-bf65-27428a66e07f-kube-api-access-944mw\" (UniqueName: \"kubernetes.io/projected/00ab6653-dd2e-4dd5-bf65-27428a66e07f-kube-api-access-944mw\") pod \"00ab6653-dd2e-4dd5-bf65-27428a66e07f\" (UID: \"00ab6653-dd2e-4dd5-bf65-27428a66e07f\") " Nov 04 05:06:06.850169 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:06.849978 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00ab6653-dd2e-4dd5-bf65-27428a66e07f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/00ab6653-dd2e-4dd5-bf65-27428a66e07f-metrics-certs\") pod \"00ab6653-dd2e-4dd5-bf65-27428a66e07f\" (UID: \"00ab6653-dd2e-4dd5-bf65-27428a66e07f\") " Nov 04 05:06:06.850169 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:06.850064 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00ab6653-dd2e-4dd5-bf65-27428a66e07f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00ab6653-dd2e-4dd5-bf65-27428a66e07f-maas-api-tls\") pod \"00ab6653-dd2e-4dd5-bf65-27428a66e07f\" (UID: \"00ab6653-dd2e-4dd5-bf65-27428a66e07f\") " Nov 04 05:06:06.850169 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:06.850108 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00ab6653-dd2e-4dd5-bf65-27428a66e07f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00ab6653-dd2e-4dd5-bf65-27428a66e07f-openshift-service-ca\") pod \"00ab6653-dd2e-4dd5-bf65-27428a66e07f\" (UID: \"00ab6653-dd2e-4dd5-bf65-27428a66e07f\") " Nov 04 05:06:06.850574 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:06.850545 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00ab6653-dd2e-4dd5-bf65-27428a66e07f-openshift-service-ca" pod "00ab6653-dd2e-4dd5-bf65-27428a66e07f" (UID: "00ab6653-dd2e-4dd5-bf65-27428a66e07f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:06:06.852418 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:06.852389 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00ab6653-dd2e-4dd5-bf65-27428a66e07f-maas-api-tls" pod "00ab6653-dd2e-4dd5-bf65-27428a66e07f" (UID: "00ab6653-dd2e-4dd5-bf65-27428a66e07f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:06:06.852515 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:06.852426 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00ab6653-dd2e-4dd5-bf65-27428a66e07f-metrics-certs" pod "00ab6653-dd2e-4dd5-bf65-27428a66e07f" (UID: "00ab6653-dd2e-4dd5-bf65-27428a66e07f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:06:06.852515 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:06.852439 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ab6653-dd2e-4dd5-bf65-27428a66e07f-kube-api-access-944mw" pod "00ab6653-dd2e-4dd5-bf65-27428a66e07f" (UID: "00ab6653-dd2e-4dd5-bf65-27428a66e07f"). InnerVolumeSpecName "kube-api-access-944mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:06.951460 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:06.951376 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-944mw\" (UniqueName: \"kubernetes.io/projected/00ab6653-dd2e-4dd5-bf65-27428a66e07f-kube-api-access-944mw\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:06.951460 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:06.951409 4581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00ab6653-dd2e-4dd5-bf65-27428a66e07f-metrics-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:06.951460 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:06.951419 4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00ab6653-dd2e-4dd5-bf65-27428a66e07f-maas-api-tls\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:06.951460 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:06.951427 4581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00ab6653-dd2e-4dd5-bf65-27428a66e07f-openshift-service-ca\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:07.459059 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:07.459020 4581 generic.go:356] "Generic (PLEG): container finished" podID="00ab6653-dd2e-4dd5-bf65-27428a66e07f" containerID="91bbdad1580664b2a96d124b863ad28e1720be1e8a39024517af8bdcf5742f5e" exitCode=0 Nov 04 05:06:07.459594 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:07.459070 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" event={"ID":"00ab6653-dd2e-4dd5-bf65-27428a66e07f","Type":"ContainerDied","Data":"91bbdad1580664b2a96d124b863ad28e1720be1e8a39024517af8bdcf5742f5e"} Nov 04 05:06:07.459594 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:07.459099 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" Nov 04 05:06:07.459594 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:07.459135 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v" event={"ID":"00ab6653-dd2e-4dd5-bf65-27428a66e07f","Type":"ContainerDied","Data":"ce017997e493c7e05b40fc032fe20b385ad182ff2ba9b644dc6fbda55f3391bc"} Nov 04 05:06:07.459594 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:07.459164 4581 scope.go:122] "RemoveContainer" containerID="91bbdad1580664b2a96d124b863ad28e1720be1e8a39024517af8bdcf5742f5e" Nov 04 05:06:07.468364 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:07.468342 4581 scope.go:122] "RemoveContainer" containerID="91bbdad1580664b2a96d124b863ad28e1720be1e8a39024517af8bdcf5742f5e" Nov 04 05:06:07.468689 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:06:07.468664 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91bbdad1580664b2a96d124b863ad28e1720be1e8a39024517af8bdcf5742f5e\": container with ID starting with 91bbdad1580664b2a96d124b863ad28e1720be1e8a39024517af8bdcf5742f5e not found: ID does not exist" containerID="91bbdad1580664b2a96d124b863ad28e1720be1e8a39024517af8bdcf5742f5e" Nov 04 05:06:07.468758 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:07.468697 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91bbdad1580664b2a96d124b863ad28e1720be1e8a39024517af8bdcf5742f5e"} err="failed to get container status \"91bbdad1580664b2a96d124b863ad28e1720be1e8a39024517af8bdcf5742f5e\": rpc error: code = NotFound desc = could not find container \"91bbdad1580664b2a96d124b863ad28e1720be1e8a39024517af8bdcf5742f5e\": container with ID starting with 91bbdad1580664b2a96d124b863ad28e1720be1e8a39024517af8bdcf5742f5e not found: ID does not exist" Nov 04 05:06:07.624093 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:07.624059 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v"] Nov 04 05:06:07.627144 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:07.627116 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-cedfca-74c47f44f5-48z2v"] Nov 04 05:06:07.987167 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:07.987046 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc" podUID="bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e" containerName="payload-pre-processing" containerID="cri-o://8bbb1945aeb28d145c6a8da934685d05df7afef6f125baa1cc83be31a53b7ecc" gracePeriod=25 Nov 04 05:06:08.174537 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.174505 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:08.415777 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.415740 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" podUID="4977f3bb-f6f9-4fc1-9934-9e02d5c2e937" containerName="main" containerID="cri-o://bcf5f2ee14e657c11c4eef6e4544329ae23a3333494f7db7c49e9088d96d8a9a" gracePeriod=45 Nov 04 05:06:08.464515 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.464480 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" podUID="44643611-a133-437f-87dd-d5413adb495e" containerName="main" containerID="cri-o://09cbe4b827fab8a6effbc45f762d8cb373e01cff7e983b6a68fecdf953e7f46b" gracePeriod=45 Nov 04 05:06:08.708884 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.708793 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:08.752234 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.752205 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 05:06:08.755460 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.755437 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:06:08.766920 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.766889 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-model-cache\") pod \"44643611-a133-437f-87dd-d5413adb495e\" (UID: \"44643611-a133-437f-87dd-d5413adb495e\") " Nov 04 05:06:08.767082 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.766935 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-dshm\") pod \"44643611-a133-437f-87dd-d5413adb495e\" (UID: \"44643611-a133-437f-87dd-d5413adb495e\") " Nov 04 05:06:08.767082 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.766970 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-kube-api-access-5hvtc\" (UniqueName: \"kubernetes.io/projected/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-kube-api-access-5hvtc\") pod \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\" (UID: \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\") " Nov 04 05:06:08.767202 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.767083 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44643611-a133-437f-87dd-d5413adb495e-tls-certs\" (UniqueName: \"kubernetes.io/secret/44643611-a133-437f-87dd-d5413adb495e-tls-certs\") pod \"44643611-a133-437f-87dd-d5413adb495e\" (UID: \"44643611-a133-437f-87dd-d5413adb495e\") " Nov 04 05:06:08.767202 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.767124 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-tmp-dir\") pod \"44643611-a133-437f-87dd-d5413adb495e\" (UID: \"44643611-a133-437f-87dd-d5413adb495e\") " Nov 04 05:06:08.767202 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.767164 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-tls-certs\" (UniqueName: \"kubernetes.io/secret/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-tls-certs\") pod \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\" (UID: \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\") " Nov 04 05:06:08.767377 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.767208 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44643611-a133-437f-87dd-d5413adb495e-kube-api-access-rrtg2\" (UniqueName: \"kubernetes.io/projected/44643611-a133-437f-87dd-d5413adb495e-kube-api-access-rrtg2\") pod \"44643611-a133-437f-87dd-d5413adb495e\" (UID: \"44643611-a133-437f-87dd-d5413adb495e\") " Nov 04 05:06:08.767377 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.767237 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-model-cache\") pod \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\" (UID: \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\") " Nov 04 05:06:08.767377 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.767264 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-home\" (UniqueName: \"kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-home\") pod \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\" (UID: \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\") " Nov 04 05:06:08.767377 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.767223 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-model-cache" pod "44643611-a133-437f-87dd-d5413adb495e" (UID: "44643611-a133-437f-87dd-d5413adb495e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:08.767377 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.767299 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-dshm\") pod \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\" (UID: \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\") " Nov 04 05:06:08.767377 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.767335 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-tmp-dir\") pod \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\" (UID: \"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937\") " Nov 04 05:06:08.767377 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.767372 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-home\" (UniqueName: \"kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-home\") pod \"44643611-a133-437f-87dd-d5413adb495e\" (UID: \"44643611-a133-437f-87dd-d5413adb495e\") " Nov 04 05:06:08.767799 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.767778 4581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-model-cache\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:08.768256 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.768232 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-home" pod "44643611-a133-437f-87dd-d5413adb495e" (UID: "44643611-a133-437f-87dd-d5413adb495e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:08.768571 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.768542 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-tmp-dir" pod "44643611-a133-437f-87dd-d5413adb495e" (UID: "44643611-a133-437f-87dd-d5413adb495e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:08.769678 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.768902 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-home" pod "4977f3bb-f6f9-4fc1-9934-9e02d5c2e937" (UID: "4977f3bb-f6f9-4fc1-9934-9e02d5c2e937"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:08.769678 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.769124 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-model-cache" pod "4977f3bb-f6f9-4fc1-9934-9e02d5c2e937" (UID: "4977f3bb-f6f9-4fc1-9934-9e02d5c2e937"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:08.769678 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.769291 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-tmp-dir" pod "4977f3bb-f6f9-4fc1-9934-9e02d5c2e937" (UID: "4977f3bb-f6f9-4fc1-9934-9e02d5c2e937"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:08.774455 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.774422 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44643611-a133-437f-87dd-d5413adb495e-kube-api-access-rrtg2" pod "44643611-a133-437f-87dd-d5413adb495e" (UID: "44643611-a133-437f-87dd-d5413adb495e"). InnerVolumeSpecName "kube-api-access-rrtg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:08.774620 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.774546 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-tls-certs" pod "4977f3bb-f6f9-4fc1-9934-9e02d5c2e937" (UID: "4977f3bb-f6f9-4fc1-9934-9e02d5c2e937"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:06:08.775099 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.775065 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44643611-a133-437f-87dd-d5413adb495e-tls-certs" pod "44643611-a133-437f-87dd-d5413adb495e" (UID: "44643611-a133-437f-87dd-d5413adb495e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:06:08.775253 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.775222 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-kube-api-access-5hvtc" pod "4977f3bb-f6f9-4fc1-9934-9e02d5c2e937" (UID: "4977f3bb-f6f9-4fc1-9934-9e02d5c2e937"). InnerVolumeSpecName "kube-api-access-5hvtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:08.775558 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.775518 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-dshm" pod "44643611-a133-437f-87dd-d5413adb495e" (UID: "44643611-a133-437f-87dd-d5413adb495e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:08.775718 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.775633 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-dshm" pod "4977f3bb-f6f9-4fc1-9934-9e02d5c2e937" (UID: "4977f3bb-f6f9-4fc1-9934-9e02d5c2e937"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:08.868910 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.868849 4581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-tmp-dir\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:08.868910 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.868880 4581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-tls-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:08.868910 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.868896 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrtg2\" (UniqueName: \"kubernetes.io/projected/44643611-a133-437f-87dd-d5413adb495e-kube-api-access-rrtg2\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:08.868910 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.868918 4581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-model-cache\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:08.869308 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.868931 4581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-home\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:08.869308 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.868945 4581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-dshm\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:08.869308 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.868956 4581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-tmp-dir\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:08.869308 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.868967 4581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-home\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:08.869308 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.868978 4581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44643611-a133-437f-87dd-d5413adb495e-dshm\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:08.869308 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.869005 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hvtc\" (UniqueName: \"kubernetes.io/projected/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937-kube-api-access-5hvtc\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:08.869308 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.869017 4581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44643611-a133-437f-87dd-d5413adb495e-tls-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:08.892109 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:08.892070 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:09.230371 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:09.230332 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00ab6653-dd2e-4dd5-bf65-27428a66e07f" path="/var/lib/kubelet/pods/00ab6653-dd2e-4dd5-bf65-27428a66e07f/volumes" Nov 04 05:06:09.414081 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:09.414040 4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc" podUID="bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 05:06:09.474250 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:09.474221 4581 generic.go:356] "Generic (PLEG): container finished" podID="44643611-a133-437f-87dd-d5413adb495e" containerID="09cbe4b827fab8a6effbc45f762d8cb373e01cff7e983b6a68fecdf953e7f46b" exitCode=0 Nov 04 05:06:09.474698 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:09.474267 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" Nov 04 05:06:09.474698 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:09.474315 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" event={"ID":"44643611-a133-437f-87dd-d5413adb495e","Type":"ContainerDied","Data":"09cbe4b827fab8a6effbc45f762d8cb373e01cff7e983b6a68fecdf953e7f46b"} Nov 04 05:06:09.474698 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:09.474357 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc" event={"ID":"44643611-a133-437f-87dd-d5413adb495e","Type":"ContainerDied","Data":"91f36421ff0abc28abbf134dd82abb68488eb33562b86ab4068b90a98ddb9ad0"} Nov 04 05:06:09.474698 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:09.474379 4581 scope.go:122] "RemoveContainer" containerID="09cbe4b827fab8a6effbc45f762d8cb373e01cff7e983b6a68fecdf953e7f46b" Nov 04 05:06:09.475958 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:09.475933 4581 generic.go:356] "Generic (PLEG): container finished" podID="4977f3bb-f6f9-4fc1-9934-9e02d5c2e937" containerID="bcf5f2ee14e657c11c4eef6e4544329ae23a3333494f7db7c49e9088d96d8a9a" exitCode=0 Nov 04 05:06:09.476103 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:09.475969 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" Nov 04 05:06:09.476103 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:09.476071 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" event={"ID":"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937","Type":"ContainerDied","Data":"bcf5f2ee14e657c11c4eef6e4544329ae23a3333494f7db7c49e9088d96d8a9a"} Nov 04 05:06:09.476217 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:09.476105 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl" event={"ID":"4977f3bb-f6f9-4fc1-9934-9e02d5c2e937","Type":"ContainerDied","Data":"b261adb8c3b5be97e6575fb82ac2a2355e1fcff1c1a6c73e4448efaf869d2278"} Nov 04 05:06:09.483855 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:09.483836 4581 scope.go:122] "RemoveContainer" containerID="09cbe4b827fab8a6effbc45f762d8cb373e01cff7e983b6a68fecdf953e7f46b" Nov 04 05:06:09.484207 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:06:09.484186 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09cbe4b827fab8a6effbc45f762d8cb373e01cff7e983b6a68fecdf953e7f46b\": container with ID starting with 09cbe4b827fab8a6effbc45f762d8cb373e01cff7e983b6a68fecdf953e7f46b not found: ID does not exist" containerID="09cbe4b827fab8a6effbc45f762d8cb373e01cff7e983b6a68fecdf953e7f46b" Nov 04 05:06:09.484301 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:09.484219 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09cbe4b827fab8a6effbc45f762d8cb373e01cff7e983b6a68fecdf953e7f46b"} err="failed to get container status \"09cbe4b827fab8a6effbc45f762d8cb373e01cff7e983b6a68fecdf953e7f46b\": rpc error: code = NotFound desc = could not find container \"09cbe4b827fab8a6effbc45f762d8cb373e01cff7e983b6a68fecdf953e7f46b\": container with ID starting with 09cbe4b827fab8a6effbc45f762d8cb373e01cff7e983b6a68fecdf953e7f46b not found: ID does not exist" Nov 04 05:06:09.484301 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:09.484244 4581 scope.go:122] "RemoveContainer" containerID="bcf5f2ee14e657c11c4eef6e4544329ae23a3333494f7db7c49e9088d96d8a9a" Nov 04 05:06:09.495094 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:09.495069 4581 scope.go:122] "RemoveContainer" containerID="bcf5f2ee14e657c11c4eef6e4544329ae23a3333494f7db7c49e9088d96d8a9a" Nov 04 05:06:09.495601 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:06:09.495581 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf5f2ee14e657c11c4eef6e4544329ae23a3333494f7db7c49e9088d96d8a9a\": container with ID starting with bcf5f2ee14e657c11c4eef6e4544329ae23a3333494f7db7c49e9088d96d8a9a not found: ID does not exist" containerID="bcf5f2ee14e657c11c4eef6e4544329ae23a3333494f7db7c49e9088d96d8a9a" Nov 04 05:06:09.495691 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:09.495609 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf5f2ee14e657c11c4eef6e4544329ae23a3333494f7db7c49e9088d96d8a9a"} err="failed to get container status \"bcf5f2ee14e657c11c4eef6e4544329ae23a3333494f7db7c49e9088d96d8a9a\": rpc error: code = NotFound desc = could not find container \"bcf5f2ee14e657c11c4eef6e4544329ae23a3333494f7db7c49e9088d96d8a9a\": container with ID starting with bcf5f2ee14e657c11c4eef6e4544329ae23a3333494f7db7c49e9088d96d8a9a not found: ID does not exist" Nov 04 05:06:09.640525 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:09.640491 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl"] Nov 04 05:06:09.690593 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:09.690559 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-4cce33/distinct-2-w5-4cce33-kserve-5fc5bb9db8-t7nkl"] Nov 04 05:06:09.755520 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:09.755442 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc"] Nov 04 05:06:09.764511 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:09.764481 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-4cce33/simulator-w5-4cce33-kserve-6549678c49-vhstc"] Nov 04 05:06:11.231257 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:11.231224 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44643611-a133-437f-87dd-d5413adb495e" path="/var/lib/kubelet/pods/44643611-a133-437f-87dd-d5413adb495e/volumes" Nov 04 05:06:11.231666 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:11.231653 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4977f3bb-f6f9-4fc1-9934-9e02d5c2e937" path="/var/lib/kubelet/pods/4977f3bb-f6f9-4fc1-9934-9e02d5c2e937/volumes" Nov 04 05:06:12.394258 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:12.394222 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj"] Nov 04 05:06:12.394770 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:12.394434 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" podUID="373125b2-53a5-449e-9fd5-a79c5fcd06fa" containerName="maas-api" containerID="cri-o://e24108dc7caade34ce24169d49b59eda4c01efc8ed0d2e1d6eb3ecc44a21d50e" gracePeriod=30 Nov 04 05:06:12.483280 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:12.478746 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:13.514067 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:13.513955 4581 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" podUID="373125b2-53a5-449e-9fd5-a79c5fcd06fa" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.48:8443/health\": dial tcp 10.134.0.48:8443: connect: connection refused" Nov 04 05:06:13.645283 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:13.645259 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" Nov 04 05:06:13.720647 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:13.720621 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/373125b2-53a5-449e-9fd5-a79c5fcd06fa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/373125b2-53a5-449e-9fd5-a79c5fcd06fa-openshift-service-ca\") pod \"373125b2-53a5-449e-9fd5-a79c5fcd06fa\" (UID: \"373125b2-53a5-449e-9fd5-a79c5fcd06fa\") " Nov 04 05:06:13.720794 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:13.720681 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/373125b2-53a5-449e-9fd5-a79c5fcd06fa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/373125b2-53a5-449e-9fd5-a79c5fcd06fa-metrics-certs\") pod \"373125b2-53a5-449e-9fd5-a79c5fcd06fa\" (UID: \"373125b2-53a5-449e-9fd5-a79c5fcd06fa\") " Nov 04 05:06:13.720794 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:13.720724 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/373125b2-53a5-449e-9fd5-a79c5fcd06fa-kube-api-access-jbnxl\" (UniqueName: \"kubernetes.io/projected/373125b2-53a5-449e-9fd5-a79c5fcd06fa-kube-api-access-jbnxl\") pod \"373125b2-53a5-449e-9fd5-a79c5fcd06fa\" (UID: \"373125b2-53a5-449e-9fd5-a79c5fcd06fa\") " Nov 04 05:06:13.720875 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:13.720799 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/373125b2-53a5-449e-9fd5-a79c5fcd06fa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/373125b2-53a5-449e-9fd5-a79c5fcd06fa-maas-api-tls\") pod \"373125b2-53a5-449e-9fd5-a79c5fcd06fa\" (UID: \"373125b2-53a5-449e-9fd5-a79c5fcd06fa\") " Nov 04 05:06:13.721047 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:13.721015 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373125b2-53a5-449e-9fd5-a79c5fcd06fa-openshift-service-ca" pod "373125b2-53a5-449e-9fd5-a79c5fcd06fa" (UID: "373125b2-53a5-449e-9fd5-a79c5fcd06fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:06:13.721235 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:13.721219 4581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/373125b2-53a5-449e-9fd5-a79c5fcd06fa-openshift-service-ca\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:13.723043 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:13.722973 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/373125b2-53a5-449e-9fd5-a79c5fcd06fa-maas-api-tls" pod "373125b2-53a5-449e-9fd5-a79c5fcd06fa" (UID: "373125b2-53a5-449e-9fd5-a79c5fcd06fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:06:13.723043 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:13.723007 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/373125b2-53a5-449e-9fd5-a79c5fcd06fa-metrics-certs" pod "373125b2-53a5-449e-9fd5-a79c5fcd06fa" (UID: "373125b2-53a5-449e-9fd5-a79c5fcd06fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:06:13.723223 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:13.723199 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373125b2-53a5-449e-9fd5-a79c5fcd06fa-kube-api-access-jbnxl" pod "373125b2-53a5-449e-9fd5-a79c5fcd06fa" (UID: "373125b2-53a5-449e-9fd5-a79c5fcd06fa"). InnerVolumeSpecName "kube-api-access-jbnxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:13.822099 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:13.822075 4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/373125b2-53a5-449e-9fd5-a79c5fcd06fa-maas-api-tls\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:13.822099 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:13.822097 4581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/373125b2-53a5-449e-9fd5-a79c5fcd06fa-metrics-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:13.822297 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:13.822108 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbnxl\" (UniqueName: \"kubernetes.io/projected/373125b2-53a5-449e-9fd5-a79c5fcd06fa-kube-api-access-jbnxl\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:14.015008 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:14.014915 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:14.516495 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:14.516448 4581 generic.go:356] "Generic (PLEG): container finished" podID="373125b2-53a5-449e-9fd5-a79c5fcd06fa" containerID="e24108dc7caade34ce24169d49b59eda4c01efc8ed0d2e1d6eb3ecc44a21d50e" exitCode=0 Nov 04 05:06:14.516906 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:14.516511 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" event={"ID":"373125b2-53a5-449e-9fd5-a79c5fcd06fa","Type":"ContainerDied","Data":"e24108dc7caade34ce24169d49b59eda4c01efc8ed0d2e1d6eb3ecc44a21d50e"} Nov 04 05:06:14.516906 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:14.516518 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" Nov 04 05:06:14.516906 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:14.516590 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj" event={"ID":"373125b2-53a5-449e-9fd5-a79c5fcd06fa","Type":"ContainerDied","Data":"525dde858f827c7bb7e9db8f8360f1a81d25934064daa87a71ba230f02b2aa64"} Nov 04 05:06:14.516906 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:14.516617 4581 scope.go:122] "RemoveContainer" containerID="e24108dc7caade34ce24169d49b59eda4c01efc8ed0d2e1d6eb3ecc44a21d50e" Nov 04 05:06:14.526528 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:14.526511 4581 scope.go:122] "RemoveContainer" containerID="e24108dc7caade34ce24169d49b59eda4c01efc8ed0d2e1d6eb3ecc44a21d50e" Nov 04 05:06:14.526794 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:06:14.526778 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e24108dc7caade34ce24169d49b59eda4c01efc8ed0d2e1d6eb3ecc44a21d50e\": container with ID starting with e24108dc7caade34ce24169d49b59eda4c01efc8ed0d2e1d6eb3ecc44a21d50e not found: ID does not exist" containerID="e24108dc7caade34ce24169d49b59eda4c01efc8ed0d2e1d6eb3ecc44a21d50e" Nov 04 05:06:14.526836 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:14.526803 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e24108dc7caade34ce24169d49b59eda4c01efc8ed0d2e1d6eb3ecc44a21d50e"} err="failed to get container status \"e24108dc7caade34ce24169d49b59eda4c01efc8ed0d2e1d6eb3ecc44a21d50e\": rpc error: code = NotFound desc = could not find container \"e24108dc7caade34ce24169d49b59eda4c01efc8ed0d2e1d6eb3ecc44a21d50e\": container with ID starting with e24108dc7caade34ce24169d49b59eda4c01efc8ed0d2e1d6eb3ecc44a21d50e not found: ID does not exist" Nov 04 05:06:14.682127 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:14.682092 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj"] Nov 04 05:06:14.732344 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:14.732316 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-4cce33-95bc6f45-tpdjj"] Nov 04 05:06:15.229270 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:15.229230 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="373125b2-53a5-449e-9fd5-a79c5fcd06fa" path="/var/lib/kubelet/pods/373125b2-53a5-449e-9fd5-a79c5fcd06fa/volumes" Nov 04 05:06:15.806826 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:15.806743 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:15.928748 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:15.928715 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:16.454349 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:16.454316 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:17.886586 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:06:17.886551 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 05:06:19.414376 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:19.414343 4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc" podUID="bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 05:06:19.969274 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:19.969236 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:20.101714 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:20.101676 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:21.937454 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.937419 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d"] Nov 04 05:06:21.937920 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.937906 4581 cpu_manager.go:414] "containerMap: removing container" podUID="373125b2-53a5-449e-9fd5-a79c5fcd06fa" containerName="maas-api" Nov 04 05:06:21.938061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.937921 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="373125b2-53a5-449e-9fd5-a79c5fcd06fa" containerName="maas-api" Nov 04 05:06:21.938061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.937933 4581 cpu_manager.go:414] "containerMap: removing container" podUID="00ab6653-dd2e-4dd5-bf65-27428a66e07f" containerName="maas-api" Nov 04 05:06:21.938061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.937938 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00ab6653-dd2e-4dd5-bf65-27428a66e07f" containerName="maas-api" Nov 04 05:06:21.938061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.937950 4581 cpu_manager.go:414] "containerMap: removing container" podUID="44643611-a133-437f-87dd-d5413adb495e" containerName="main" Nov 04 05:06:21.938061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.937955 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44643611-a133-437f-87dd-d5413adb495e" containerName="main" Nov 04 05:06:21.938061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.937963 4581 cpu_manager.go:414] "containerMap: removing container" podUID="1ce6a777-2cfb-4cf5-905a-209d8d82f331" containerName="istio-proxy" Nov 04 05:06:21.938061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.937971 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce6a777-2cfb-4cf5-905a-209d8d82f331" containerName="istio-proxy" Nov 04 05:06:21.938061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.937998 4581 cpu_manager.go:414] "containerMap: removing container" podUID="2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2" containerName="istio-proxy" Nov 04 05:06:21.938061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.938004 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2" containerName="istio-proxy" Nov 04 05:06:21.938061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.938010 4581 cpu_manager.go:414] "containerMap: removing container" podUID="4977f3bb-f6f9-4fc1-9934-9e02d5c2e937" containerName="main" Nov 04 05:06:21.938061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.938014 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4977f3bb-f6f9-4fc1-9934-9e02d5c2e937" containerName="main" Nov 04 05:06:21.938061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.938021 4581 cpu_manager.go:414] "containerMap: removing container" podUID="f9ac68d4-0a7a-47aa-afeb-803dfc3f362a" containerName="istio-proxy" Nov 04 05:06:21.938061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.938025 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9ac68d4-0a7a-47aa-afeb-803dfc3f362a" containerName="istio-proxy" Nov 04 05:06:21.938061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.938034 4581 cpu_manager.go:414] "containerMap: removing container" podUID="9414f040-9632-4245-b1c3-b00b229cb75a" containerName="main" Nov 04 05:06:21.938061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.938039 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9414f040-9632-4245-b1c3-b00b229cb75a" containerName="main" Nov 04 05:06:21.938502 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.938113 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce6a777-2cfb-4cf5-905a-209d8d82f331" containerName="istio-proxy" Nov 04 05:06:21.938502 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.938122 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="373125b2-53a5-449e-9fd5-a79c5fcd06fa" containerName="maas-api" Nov 04 05:06:21.938502 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.938130 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="00ab6653-dd2e-4dd5-bf65-27428a66e07f" containerName="maas-api" Nov 04 05:06:21.938502 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.938140 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="44643611-a133-437f-87dd-d5413adb495e" containerName="main" Nov 04 05:06:21.938502 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.938148 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ed81f25-e6fc-4a0c-a631-3ac69d7d64d2" containerName="istio-proxy" Nov 04 05:06:21.938502 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.938154 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="4977f3bb-f6f9-4fc1-9934-9e02d5c2e937" containerName="main" Nov 04 05:06:21.938502 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.938159 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9ac68d4-0a7a-47aa-afeb-803dfc3f362a" containerName="istio-proxy" Nov 04 05:06:21.938502 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.938166 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="9414f040-9632-4245-b1c3-b00b229cb75a" containerName="main" Nov 04 05:06:21.943026 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.942980 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:21.985815 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.985770 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:21.987754 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.987733 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9"] Nov 04 05:06:21.991200 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:21.991182 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.001669 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.001650 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wl4z\" (UniqueName: \"kubernetes.io/projected/2b61c48c-bc70-4cc1-977f-494f8f637943-kube-api-access-8wl4z\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.001756 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.001677 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-workload-certs\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.001756 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.001710 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-token\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.001874 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.001854 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-workload-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.001913 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.001892 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-podinfo\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.001949 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.001926 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-envoy\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.001981 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.001954 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-credential-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.002044 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.002011 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b61c48c-bc70-4cc1-977f-494f8f637943-istiod-ca-cert\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.002077 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.002048 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-data\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.038206 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.038175 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d"] Nov 04 05:06:22.092599 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.092566 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9"] Nov 04 05:06:22.102745 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.102717 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d77404-f335-45d1-8335-9a725c555979-istio-podinfo\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.102895 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.102769 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-data\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.102895 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.102830 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f75gj\" (UniqueName: \"kubernetes.io/projected/37d77404-f335-45d1-8335-9a725c555979-kube-api-access-f75gj\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.102895 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.102872 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d77404-f335-45d1-8335-9a725c555979-istiod-ca-cert\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.103074 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.102906 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-credential-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.103074 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.103028 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wl4z\" (UniqueName: \"kubernetes.io/projected/2b61c48c-bc70-4cc1-977f-494f8f637943-kube-api-access-8wl4z\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.103074 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.103052 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-workload-certs\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.103189 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.103104 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-token\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.103189 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.103131 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d77404-f335-45d1-8335-9a725c555979-istio-token\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.103189 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.103161 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-workload-certs\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.103189 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.103178 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-data\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.103373 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.103201 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-workload-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.103373 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.103264 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-podinfo\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.103373 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.103314 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-workload-certs\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.103484 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.103390 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-istio-envoy\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.103484 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.103449 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-envoy\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.103549 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.103487 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-workload-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.103549 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.103526 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-credential-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.103630 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.103557 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-istio-data\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.103630 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.103578 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-workload-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.103748 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.103728 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b61c48c-bc70-4cc1-977f-494f8f637943-istiod-ca-cert\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.103817 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.103803 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-credential-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.104500 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.104477 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b61c48c-bc70-4cc1-977f-494f8f637943-istiod-ca-cert\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.106442 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.106414 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-podinfo\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.106569 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.106546 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-envoy\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.161449 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.161420 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wl4z\" (UniqueName: \"kubernetes.io/projected/2b61c48c-bc70-4cc1-977f-494f8f637943-kube-api-access-8wl4z\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.161739 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.161719 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-token\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.172847 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.172815 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:22.204859 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.204787 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d77404-f335-45d1-8335-9a725c555979-istio-token\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.204859 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.204827 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-workload-certs\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.205061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.204873 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-istio-envoy\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.205061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.204924 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-workload-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.205061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.204960 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-istio-data\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.205061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.205040 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d77404-f335-45d1-8335-9a725c555979-istio-podinfo\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.205270 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.205081 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f75gj\" (UniqueName: \"kubernetes.io/projected/37d77404-f335-45d1-8335-9a725c555979-kube-api-access-f75gj\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.205270 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.205122 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d77404-f335-45d1-8335-9a725c555979-istiod-ca-cert\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.205270 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.205154 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-credential-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.205429 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.205287 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-workload-certs\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.205429 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.205350 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-workload-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.205537 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.205426 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-istio-data\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.205537 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.205527 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-credential-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.205899 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.205879 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d77404-f335-45d1-8335-9a725c555979-istiod-ca-cert\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.207244 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.207222 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-istio-envoy\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.207544 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.207526 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d77404-f335-45d1-8335-9a725c555979-istio-podinfo\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.256182 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.256159 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:22.261953 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.261934 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d77404-f335-45d1-8335-9a725c555979-istio-token\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.262177 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.262160 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f75gj\" (UniqueName: \"kubernetes.io/projected/37d77404-f335-45d1-8335-9a725c555979-kube-api-access-f75gj\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.303175 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.303142 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:22.439953 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.439924 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:22.572792 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.572721 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd"] Nov 04 05:06:22.588376 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.588344 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.631871 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.631841 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd"] Nov 04 05:06:22.686493 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.686467 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d"] Nov 04 05:06:22.687868 ip-10-0-141-132 kubenswrapper[4581]: W1104 05:06:22.687835 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b61c48c_bc70_4cc1_977f_494f8f637943.slice/crio-d9120127057c13863676e6d92be2a6ee0995874cbcb6b1f675a3ab6a9aab7745 WatchSource:0}: Error finding container d9120127057c13863676e6d92be2a6ee0995874cbcb6b1f675a3ab6a9aab7745: Status 404 returned error can't find the container with id d9120127057c13863676e6d92be2a6ee0995874cbcb6b1f675a3ab6a9aab7745 Nov 04 05:06:22.690249 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.690212 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:06:22.690340 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.690276 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:06:22.690340 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.690303 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:06:22.710498 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.710467 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-istio-data\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.710614 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.710502 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0446660f-7639-44c2-9372-17dca64e673d-istio-podinfo\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.710614 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.710519 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0446660f-7639-44c2-9372-17dca64e673d-istiod-ca-cert\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.710614 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.710582 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-workload-socket\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.710781 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.710674 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0446660f-7639-44c2-9372-17dca64e673d-istio-token\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.710781 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.710695 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-istio-envoy\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.710886 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.710778 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-credential-socket\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.710886 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.710804 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-workload-certs\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.711009 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.710882 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6827d\" (UniqueName: \"kubernetes.io/projected/0446660f-7639-44c2-9372-17dca64e673d-kube-api-access-6827d\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.735192 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.735166 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9"] Nov 04 05:06:22.737837 ip-10-0-141-132 kubenswrapper[4581]: W1104 05:06:22.737807 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d77404_f335_45d1_8335_9a725c555979.slice/crio-da8a571f7c1f8295a66067289ee526395792c04c3dfff8be048e274cdd0e8869 WatchSource:0}: Error finding container da8a571f7c1f8295a66067289ee526395792c04c3dfff8be048e274cdd0e8869: Status 404 returned error can't find the container with id da8a571f7c1f8295a66067289ee526395792c04c3dfff8be048e274cdd0e8869 Nov 04 05:06:22.739859 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.739826 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:06:22.739956 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.739903 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:06:22.739956 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.739950 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:06:22.812375 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.812350 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-credential-socket\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.812528 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.812385 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-workload-certs\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.812528 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.812454 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6827d\" (UniqueName: \"kubernetes.io/projected/0446660f-7639-44c2-9372-17dca64e673d-kube-api-access-6827d\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.812528 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.812519 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-istio-data\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.812703 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.812535 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0446660f-7639-44c2-9372-17dca64e673d-istio-podinfo\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.812703 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.812550 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0446660f-7639-44c2-9372-17dca64e673d-istiod-ca-cert\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.812703 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.812568 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-workload-socket\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.812703 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.812592 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0446660f-7639-44c2-9372-17dca64e673d-istio-token\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.812703 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.812612 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-istio-envoy\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.812945 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.812758 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-credential-socket\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.813287 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.813037 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-workload-certs\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.813287 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.813163 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-istio-data\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.813641 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.813544 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-workload-socket\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.814209 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.814185 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0446660f-7639-44c2-9372-17dca64e673d-istiod-ca-cert\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.815032 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.815007 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-istio-envoy\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.815614 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.815420 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0446660f-7639-44c2-9372-17dca64e673d-istio-podinfo\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.870573 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.870536 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0446660f-7639-44c2-9372-17dca64e673d-istio-token\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.870573 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.870552 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6827d\" (UniqueName: \"kubernetes.io/projected/0446660f-7639-44c2-9372-17dca64e673d-kube-api-access-6827d\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.904089 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.904042 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:22.949218 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.949171 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776"] Nov 04 05:06:22.950397 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.950220 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" podUID="c1c643f4-9bbd-48f1-a6eb-819bc48c352e" containerName="istio-proxy" containerID="cri-o://d698b2df7c0e3c232d5720a9e54c440c0fa9f919a885411ada3a86b28433a5af" gracePeriod=30 Nov 04 05:06:22.952548 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.952521 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd"] Nov 04 05:06:22.970568 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:22.970474 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:23.326100 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:23.326067 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd"] Nov 04 05:06:23.328455 ip-10-0-141-132 kubenswrapper[4581]: W1104 05:06:23.328427 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0446660f_7639_44c2_9372_17dca64e673d.slice/crio-6324348a87dc92d50d6b79ea0ec9d1040dba27542e7700450d4fb497333e1fea WatchSource:0}: Error finding container 6324348a87dc92d50d6b79ea0ec9d1040dba27542e7700450d4fb497333e1fea: Status 404 returned error can't find the container with id 6324348a87dc92d50d6b79ea0ec9d1040dba27542e7700450d4fb497333e1fea Nov 04 05:06:23.331090 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:23.331032 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:06:23.331179 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:23.331159 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:06:23.331246 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:23.331230 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:06:23.611661 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:23.611365 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" event={"ID":"2b61c48c-bc70-4cc1-977f-494f8f637943","Type":"ContainerStarted","Data":"fb83921215a0f0c8505251d7f0f311f6bf809da333f5661cede46952a937a077"} Nov 04 05:06:23.611661 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:23.611437 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" event={"ID":"2b61c48c-bc70-4cc1-977f-494f8f637943","Type":"ContainerStarted","Data":"d9120127057c13863676e6d92be2a6ee0995874cbcb6b1f675a3ab6a9aab7745"} Nov 04 05:06:23.614064 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:23.614039 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" event={"ID":"37d77404-f335-45d1-8335-9a725c555979","Type":"ContainerStarted","Data":"0e7137c2769281f3885b0737fecd776c14749ffb0ed81fa4324f4d763b8c9495"} Nov 04 05:06:23.614190 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:23.614083 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" event={"ID":"37d77404-f335-45d1-8335-9a725c555979","Type":"ContainerStarted","Data":"da8a571f7c1f8295a66067289ee526395792c04c3dfff8be048e274cdd0e8869"} Nov 04 05:06:23.615597 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:23.615574 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" event={"ID":"0446660f-7639-44c2-9372-17dca64e673d","Type":"ContainerStarted","Data":"49a0ec3353f786ba554ed43e094575ab2b43a722b4c794c254c1932249b6e24f"} Nov 04 05:06:23.615693 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:23.615614 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" event={"ID":"0446660f-7639-44c2-9372-17dca64e673d","Type":"ContainerStarted","Data":"6324348a87dc92d50d6b79ea0ec9d1040dba27542e7700450d4fb497333e1fea"} Nov 04 05:06:23.615787 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:23.615768 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" podUID="0446660f-7639-44c2-9372-17dca64e673d" containerName="istio-proxy" containerID="cri-o://49a0ec3353f786ba554ed43e094575ab2b43a722b4c794c254c1932249b6e24f" gracePeriod=30 Nov 04 05:06:23.732208 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:23.730612 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" podStartSLOduration=2.730597416 podStartE2EDuration="2.730597416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:06:23.729088259 +0000 UTC m=+1142.619945270" watchObservedRunningTime="2026-11-04 05:06:23.730597416 +0000 UTC m=+1142.621454507" Nov 04 05:06:23.868072 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:23.853963 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" podStartSLOduration=2.853944209 podStartE2EDuration="2.853944209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:06:23.851824616 +0000 UTC m=+1142.742681630" watchObservedRunningTime="2026-11-04 05:06:23.853944209 +0000 UTC m=+1142.744801230" Nov 04 05:06:23.994012 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:23.978515 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" podStartSLOduration=1.978492729 podStartE2EDuration="1.978492729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:06:23.975485671 +0000 UTC m=+1142.866342696" watchObservedRunningTime="2026-11-04 05:06:23.978492729 +0000 UTC m=+1142.869349752" Nov 04 05:06:24.266262 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:24.260339 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:24.266262 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:24.264801 4581 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 04 05:06:24.266262 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:24.264859 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" podUID="2b61c48c-bc70-4cc1-977f-494f8f637943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 04 05:06:24.304065 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:24.304032 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:24.305965 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:24.305920 4581 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 04 05:06:24.306069 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:24.306018 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" podUID="37d77404-f335-45d1-8335-9a725c555979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 04 05:06:25.268013 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:25.265135 4581 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 04 05:06:25.268013 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:25.265239 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" podUID="2b61c48c-bc70-4cc1-977f-494f8f637943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 04 05:06:25.304548 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:25.304514 4581 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 04 05:06:25.304714 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:25.304570 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" podUID="37d77404-f335-45d1-8335-9a725c555979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 04 05:06:26.264013 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:26.261368 4581 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 04 05:06:26.264013 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:26.261430 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" podUID="2b61c48c-bc70-4cc1-977f-494f8f637943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 04 05:06:26.304442 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:26.304412 4581 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 04 05:06:26.304927 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:26.304894 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" podUID="37d77404-f335-45d1-8335-9a725c555979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 04 05:06:27.268446 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:27.268406 4581 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 04 05:06:27.268703 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:27.268674 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" podUID="2b61c48c-bc70-4cc1-977f-494f8f637943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 04 05:06:27.309034 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:27.306434 4581 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 04 05:06:27.310250 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:27.310217 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" podUID="37d77404-f335-45d1-8335-9a725c555979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 04 05:06:28.269080 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.261377 4581 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 04 05:06:28.269080 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.261436 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" podUID="2b61c48c-bc70-4cc1-977f-494f8f637943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 04 05:06:28.310661 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.310514 4581 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 04 05:06:28.310661 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.310581 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" podUID="37d77404-f335-45d1-8335-9a725c555979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 04 05:06:28.335957 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.335931 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:06:28.409426 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.409394 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc" Nov 04 05:06:28.415409 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.415386 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-token\" (UniqueName: \"kubernetes.io/projected/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-token\") pod \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " Nov 04 05:06:28.415519 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.415435 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-envoy\") pod \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " Nov 04 05:06:28.415519 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.415470 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-data\") pod \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " Nov 04 05:06:28.415519 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.415487 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-kube-api-access-4h9hn\" (UniqueName: \"kubernetes.io/projected/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-kube-api-access-4h9hn\") pod \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " Nov 04 05:06:28.415519 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.415505 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-workload-certs\") pod \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " Nov 04 05:06:28.415734 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.415532 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-podinfo\") pod \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " Nov 04 05:06:28.415734 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.415557 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-credential-socket\") pod \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " Nov 04 05:06:28.415734 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.415604 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-workload-socket\") pod \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " Nov 04 05:06:28.415734 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.415654 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istiod-ca-cert\") pod \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\" (UID: \"c1c643f4-9bbd-48f1-a6eb-819bc48c352e\") " Nov 04 05:06:28.416124 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.416051 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-data" pod "c1c643f4-9bbd-48f1-a6eb-819bc48c352e" (UID: "c1c643f4-9bbd-48f1-a6eb-819bc48c352e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:28.416676 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.416597 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-workload-socket" pod "c1c643f4-9bbd-48f1-a6eb-819bc48c352e" (UID: "c1c643f4-9bbd-48f1-a6eb-819bc48c352e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:28.416676 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.416255 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-workload-certs" pod "c1c643f4-9bbd-48f1-a6eb-819bc48c352e" (UID: "c1c643f4-9bbd-48f1-a6eb-819bc48c352e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:28.416676 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.416253 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istiod-ca-cert" pod "c1c643f4-9bbd-48f1-a6eb-819bc48c352e" (UID: "c1c643f4-9bbd-48f1-a6eb-819bc48c352e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:06:28.416676 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.416454 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-credential-socket" pod "c1c643f4-9bbd-48f1-a6eb-819bc48c352e" (UID: "c1c643f4-9bbd-48f1-a6eb-819bc48c352e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:28.417813 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.417773 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-token" pod "c1c643f4-9bbd-48f1-a6eb-819bc48c352e" (UID: "c1c643f4-9bbd-48f1-a6eb-819bc48c352e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:28.417896 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.417858 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-podinfo" pod "c1c643f4-9bbd-48f1-a6eb-819bc48c352e" (UID: "c1c643f4-9bbd-48f1-a6eb-819bc48c352e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:06:28.418109 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.418091 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-envoy" pod "c1c643f4-9bbd-48f1-a6eb-819bc48c352e" (UID: "c1c643f4-9bbd-48f1-a6eb-819bc48c352e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:28.418376 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.418356 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-kube-api-access-4h9hn" pod "c1c643f4-9bbd-48f1-a6eb-819bc48c352e" (UID: "c1c643f4-9bbd-48f1-a6eb-819bc48c352e"). InnerVolumeSpecName "kube-api-access-4h9hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:28.517039 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.516340 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e-plugins-config-volume\") pod \"bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e\" (UID: \"bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e\") " Nov 04 05:06:28.517039 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.516470 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e-kube-api-access-zjr6n\" (UniqueName: \"kubernetes.io/projected/bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e-kube-api-access-zjr6n\") pod \"bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e\" (UID: \"bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e\") " Nov 04 05:06:28.517039 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.516849 4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-token\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:28.517039 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.516868 4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-envoy\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:28.517039 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.516883 4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-data\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:28.517039 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.516897 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h9hn\" (UniqueName: \"kubernetes.io/projected/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-kube-api-access-4h9hn\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:28.517039 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.516913 4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-workload-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:28.517039 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.516926 4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istio-podinfo\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:28.517039 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.516940 4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-credential-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:28.517039 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.516953 4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-workload-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:28.517039 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.516966 4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1c643f4-9bbd-48f1-a6eb-819bc48c352e-istiod-ca-cert\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:28.517704 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.517041 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e-plugins-config-volume" pod "bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e" (UID: "bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:06:28.519364 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.519308 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e-kube-api-access-zjr6n" pod "bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e" (UID: "bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e"). InnerVolumeSpecName "kube-api-access-zjr6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:28.618050 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.618006 4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e-plugins-config-volume\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:28.618050 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.618040 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjr6n\" (UniqueName: \"kubernetes.io/projected/bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e-kube-api-access-zjr6n\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:28.645681 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.645648 4581 generic.go:356] "Generic (PLEG): container finished" podID="c1c643f4-9bbd-48f1-a6eb-819bc48c352e" containerID="d698b2df7c0e3c232d5720a9e54c440c0fa9f919a885411ada3a86b28433a5af" exitCode=0 Nov 04 05:06:28.645835 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.645708 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" Nov 04 05:06:28.645835 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.645744 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" event={"ID":"c1c643f4-9bbd-48f1-a6eb-819bc48c352e","Type":"ContainerDied","Data":"d698b2df7c0e3c232d5720a9e54c440c0fa9f919a885411ada3a86b28433a5af"} Nov 04 05:06:28.645835 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.645808 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776" event={"ID":"c1c643f4-9bbd-48f1-a6eb-819bc48c352e","Type":"ContainerDied","Data":"f861085e25e4e094d5774491fff050f3f17c1d6db8b4aaf5539a291c88d2d2a9"} Nov 04 05:06:28.646104 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.645837 4581 scope.go:122] "RemoveContainer" containerID="d698b2df7c0e3c232d5720a9e54c440c0fa9f919a885411ada3a86b28433a5af" Nov 04 05:06:28.646890 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.646871 4581 generic.go:356] "Generic (PLEG): container finished" podID="bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e" containerID="8bbb1945aeb28d145c6a8da934685d05df7afef6f125baa1cc83be31a53b7ecc" exitCode=0 Nov 04 05:06:28.646979 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.646917 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc" event={"ID":"bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e","Type":"ContainerDied","Data":"8bbb1945aeb28d145c6a8da934685d05df7afef6f125baa1cc83be31a53b7ecc"} Nov 04 05:06:28.646979 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.646924 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc" Nov 04 05:06:28.646979 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.646945 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc" event={"ID":"bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e","Type":"ContainerDied","Data":"f27ecf91f4730b6ccf2b0d2d32448af53280b41c1a0d3382cf1f40a1cfb0a32c"} Nov 04 05:06:28.732078 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.731859 4581 scope.go:122] "RemoveContainer" containerID="d698b2df7c0e3c232d5720a9e54c440c0fa9f919a885411ada3a86b28433a5af" Nov 04 05:06:28.734121 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:06:28.733859 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d698b2df7c0e3c232d5720a9e54c440c0fa9f919a885411ada3a86b28433a5af\": container with ID starting with d698b2df7c0e3c232d5720a9e54c440c0fa9f919a885411ada3a86b28433a5af not found: ID does not exist" containerID="d698b2df7c0e3c232d5720a9e54c440c0fa9f919a885411ada3a86b28433a5af" Nov 04 05:06:28.734121 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.733902 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d698b2df7c0e3c232d5720a9e54c440c0fa9f919a885411ada3a86b28433a5af"} err="failed to get container status \"d698b2df7c0e3c232d5720a9e54c440c0fa9f919a885411ada3a86b28433a5af\": rpc error: code = NotFound desc = could not find container \"d698b2df7c0e3c232d5720a9e54c440c0fa9f919a885411ada3a86b28433a5af\": container with ID starting with d698b2df7c0e3c232d5720a9e54c440c0fa9f919a885411ada3a86b28433a5af not found: ID does not exist" Nov 04 05:06:28.734121 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.733926 4581 scope.go:122] "RemoveContainer" containerID="8bbb1945aeb28d145c6a8da934685d05df7afef6f125baa1cc83be31a53b7ecc" Nov 04 05:06:28.743251 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.743049 4581 scope.go:122] "RemoveContainer" containerID="8bbb1945aeb28d145c6a8da934685d05df7afef6f125baa1cc83be31a53b7ecc" Nov 04 05:06:28.743412 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:06:28.743389 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bbb1945aeb28d145c6a8da934685d05df7afef6f125baa1cc83be31a53b7ecc\": container with ID starting with 8bbb1945aeb28d145c6a8da934685d05df7afef6f125baa1cc83be31a53b7ecc not found: ID does not exist" containerID="8bbb1945aeb28d145c6a8da934685d05df7afef6f125baa1cc83be31a53b7ecc" Nov 04 05:06:28.743490 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.743425 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bbb1945aeb28d145c6a8da934685d05df7afef6f125baa1cc83be31a53b7ecc"} err="failed to get container status \"8bbb1945aeb28d145c6a8da934685d05df7afef6f125baa1cc83be31a53b7ecc\": rpc error: code = NotFound desc = could not find container \"8bbb1945aeb28d145c6a8da934685d05df7afef6f125baa1cc83be31a53b7ecc\": container with ID starting with 8bbb1945aeb28d145c6a8da934685d05df7afef6f125baa1cc83be31a53b7ecc not found: ID does not exist" Nov 04 05:06:28.902180 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.901146 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776"] Nov 04 05:06:28.906245 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.906218 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776"] Nov 04 05:06:28.909703 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:28.909679 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:29.016242 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.016203 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc"] Nov 04 05:06:29.018270 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.018245 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc"] Nov 04 05:06:29.021922 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.021897 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0446660f-7639-44c2-9372-17dca64e673d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0446660f-7639-44c2-9372-17dca64e673d-istiod-ca-cert\") pod \"0446660f-7639-44c2-9372-17dca64e673d\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " Nov 04 05:06:29.022061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.021940 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-workload-socket\") pod \"0446660f-7639-44c2-9372-17dca64e673d\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " Nov 04 05:06:29.022061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.022008 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0446660f-7639-44c2-9372-17dca64e673d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0446660f-7639-44c2-9372-17dca64e673d-istio-podinfo\") pod \"0446660f-7639-44c2-9372-17dca64e673d\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " Nov 04 05:06:29.022061 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.022058 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-istio-data\") pod \"0446660f-7639-44c2-9372-17dca64e673d\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " Nov 04 05:06:29.022193 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.022076 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-istio-envoy\") pod \"0446660f-7639-44c2-9372-17dca64e673d\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " Nov 04 05:06:29.022193 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.022125 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0446660f-7639-44c2-9372-17dca64e673d-istio-token\" (UniqueName: \"kubernetes.io/projected/0446660f-7639-44c2-9372-17dca64e673d-istio-token\") pod \"0446660f-7639-44c2-9372-17dca64e673d\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " Nov 04 05:06:29.022284 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.022192 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-credential-socket\") pod \"0446660f-7639-44c2-9372-17dca64e673d\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " Nov 04 05:06:29.022284 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.022210 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-workload-certs\") pod \"0446660f-7639-44c2-9372-17dca64e673d\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " Nov 04 05:06:29.022284 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.022263 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0446660f-7639-44c2-9372-17dca64e673d-kube-api-access-6827d\" (UniqueName: \"kubernetes.io/projected/0446660f-7639-44c2-9372-17dca64e673d-kube-api-access-6827d\") pod \"0446660f-7639-44c2-9372-17dca64e673d\" (UID: \"0446660f-7639-44c2-9372-17dca64e673d\") " Nov 04 05:06:29.022439 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.022328 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-workload-socket" pod "0446660f-7639-44c2-9372-17dca64e673d" (UID: "0446660f-7639-44c2-9372-17dca64e673d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:29.022439 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.022336 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0446660f-7639-44c2-9372-17dca64e673d-istiod-ca-cert" pod "0446660f-7639-44c2-9372-17dca64e673d" (UID: "0446660f-7639-44c2-9372-17dca64e673d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:06:29.022624 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.022590 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-credential-socket" pod "0446660f-7639-44c2-9372-17dca64e673d" (UID: "0446660f-7639-44c2-9372-17dca64e673d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:29.022690 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.022649 4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0446660f-7639-44c2-9372-17dca64e673d-istiod-ca-cert\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:29.022690 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.022668 4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-workload-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:29.022809 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.022715 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-istio-data" pod "0446660f-7639-44c2-9372-17dca64e673d" (UID: "0446660f-7639-44c2-9372-17dca64e673d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:29.022809 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.022730 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-workload-certs" pod "0446660f-7639-44c2-9372-17dca64e673d" (UID: "0446660f-7639-44c2-9372-17dca64e673d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:29.024682 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.024648 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-istio-envoy" pod "0446660f-7639-44c2-9372-17dca64e673d" (UID: "0446660f-7639-44c2-9372-17dca64e673d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:29.024777 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.024754 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0446660f-7639-44c2-9372-17dca64e673d-istio-podinfo" pod "0446660f-7639-44c2-9372-17dca64e673d" (UID: "0446660f-7639-44c2-9372-17dca64e673d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:06:29.025100 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.025074 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0446660f-7639-44c2-9372-17dca64e673d-istio-token" pod "0446660f-7639-44c2-9372-17dca64e673d" (UID: "0446660f-7639-44c2-9372-17dca64e673d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:29.025276 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.025257 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0446660f-7639-44c2-9372-17dca64e673d-kube-api-access-6827d" pod "0446660f-7639-44c2-9372-17dca64e673d" (UID: "0446660f-7639-44c2-9372-17dca64e673d"). InnerVolumeSpecName "kube-api-access-6827d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:29.123518 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.123482 4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0446660f-7639-44c2-9372-17dca64e673d-istio-podinfo\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:29.123518 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.123509 4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-istio-data\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:29.123518 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.123520 4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-istio-envoy\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:29.123518 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.123528 4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0446660f-7639-44c2-9372-17dca64e673d-istio-token\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:29.123893 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.123538 4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-credential-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:29.123893 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.123547 4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0446660f-7639-44c2-9372-17dca64e673d-workload-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:29.123893 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.123555 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6827d\" (UniqueName: \"kubernetes.io/projected/0446660f-7639-44c2-9372-17dca64e673d-kube-api-access-6827d\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:29.230376 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.230335 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e" path="/var/lib/kubelet/pods/bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e/volumes" Nov 04 05:06:29.230751 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.230736 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1c643f4-9bbd-48f1-a6eb-819bc48c352e" path="/var/lib/kubelet/pods/c1c643f4-9bbd-48f1-a6eb-819bc48c352e/volumes" Nov 04 05:06:29.256735 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.256702 4581 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 04 05:06:29.256866 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.256771 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" podUID="2b61c48c-bc70-4cc1-977f-494f8f637943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 04 05:06:29.304359 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.304327 4581 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 04 05:06:29.304479 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.304374 4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" podUID="37d77404-f335-45d1-8335-9a725c555979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 04 05:06:29.655731 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.655695 4581 generic.go:356] "Generic (PLEG): container finished" podID="0446660f-7639-44c2-9372-17dca64e673d" containerID="49a0ec3353f786ba554ed43e094575ab2b43a722b4c794c254c1932249b6e24f" exitCode=0 Nov 04 05:06:29.656245 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.655767 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" Nov 04 05:06:29.656245 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.655773 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" event={"ID":"0446660f-7639-44c2-9372-17dca64e673d","Type":"ContainerDied","Data":"49a0ec3353f786ba554ed43e094575ab2b43a722b4c794c254c1932249b6e24f"} Nov 04 05:06:29.656245 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.655912 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd" event={"ID":"0446660f-7639-44c2-9372-17dca64e673d","Type":"ContainerDied","Data":"6324348a87dc92d50d6b79ea0ec9d1040dba27542e7700450d4fb497333e1fea"} Nov 04 05:06:29.656245 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.655943 4581 scope.go:122] "RemoveContainer" containerID="49a0ec3353f786ba554ed43e094575ab2b43a722b4c794c254c1932249b6e24f" Nov 04 05:06:29.667063 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.667040 4581 scope.go:122] "RemoveContainer" containerID="49a0ec3353f786ba554ed43e094575ab2b43a722b4c794c254c1932249b6e24f" Nov 04 05:06:29.667406 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:06:29.667380 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49a0ec3353f786ba554ed43e094575ab2b43a722b4c794c254c1932249b6e24f\": container with ID starting with 49a0ec3353f786ba554ed43e094575ab2b43a722b4c794c254c1932249b6e24f not found: ID does not exist" containerID="49a0ec3353f786ba554ed43e094575ab2b43a722b4c794c254c1932249b6e24f" Nov 04 05:06:29.667527 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.667414 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49a0ec3353f786ba554ed43e094575ab2b43a722b4c794c254c1932249b6e24f"} err="failed to get container status \"49a0ec3353f786ba554ed43e094575ab2b43a722b4c794c254c1932249b6e24f\": rpc error: code = NotFound desc = could not find container \"49a0ec3353f786ba554ed43e094575ab2b43a722b4c794c254c1932249b6e24f\": container with ID starting with 49a0ec3353f786ba554ed43e094575ab2b43a722b4c794c254c1932249b6e24f not found: ID does not exist" Nov 04 05:06:29.826604 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.826514 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd"] Nov 04 05:06:29.829961 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:29.829929 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd"] Nov 04 05:06:30.216380 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:30.216334 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:30.334670 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:30.334622 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:30.796083 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:30.796052 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:30.797199 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:30.796895 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:30.798020 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:30.797968 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:30.812959 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:30.812927 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:30.813577 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:30.813537 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:30.814606 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:30.814577 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:31.057142 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:31.057054 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:31.230919 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:31.230886 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0446660f-7639-44c2-9372-17dca64e673d" path="/var/lib/kubelet/pods/0446660f-7639-44c2-9372-17dca64e673d/volumes" Nov 04 05:06:37.075857 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.075826 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6"] Nov 04 05:06:37.076387 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.076370 4581 cpu_manager.go:414] "containerMap: removing container" podUID="bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e" containerName="payload-pre-processing" Nov 04 05:06:37.076435 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.076389 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e" containerName="payload-pre-processing" Nov 04 05:06:37.076435 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.076400 4581 cpu_manager.go:414] "containerMap: removing container" podUID="0446660f-7639-44c2-9372-17dca64e673d" containerName="istio-proxy" Nov 04 05:06:37.076435 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.076405 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0446660f-7639-44c2-9372-17dca64e673d" containerName="istio-proxy" Nov 04 05:06:37.076435 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.076423 4581 cpu_manager.go:414] "containerMap: removing container" podUID="c1c643f4-9bbd-48f1-a6eb-819bc48c352e" containerName="istio-proxy" Nov 04 05:06:37.076435 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.076428 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c643f4-9bbd-48f1-a6eb-819bc48c352e" containerName="istio-proxy" Nov 04 05:06:37.076580 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.076491 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c643f4-9bbd-48f1-a6eb-819bc48c352e" containerName="istio-proxy" Nov 04 05:06:37.076580 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.076501 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="0446660f-7639-44c2-9372-17dca64e673d" containerName="istio-proxy" Nov 04 05:06:37.076580 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.076508 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc2f10e9-0b97-4522-bc6f-c8ac8e4ef30e" containerName="payload-pre-processing" Nov 04 05:06:37.081209 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.081192 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.106242 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.106215 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-token\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.106405 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.106251 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-data\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.106405 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.106272 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-envoy\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.106405 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.106333 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-credential-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.106578 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.106480 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-workload-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.106578 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.106514 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b6882c-5393-4adc-ae92-a45a12880e5c-istiod-ca-cert\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.106578 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.106550 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf8hk\" (UniqueName: \"kubernetes.io/projected/19b6882c-5393-4adc-ae92-a45a12880e5c-kube-api-access-sf8hk\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.106730 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.106631 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-podinfo\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.106782 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.106751 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-workload-certs\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.186512 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.186478 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6"] Nov 04 05:06:37.207770 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.207738 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-token\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.207926 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.207782 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-data\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.207926 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.207807 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-envoy\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.207926 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.207832 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-credential-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.207926 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.207864 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-workload-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.207926 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.207885 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b6882c-5393-4adc-ae92-a45a12880e5c-istiod-ca-cert\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.207926 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.207918 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf8hk\" (UniqueName: \"kubernetes.io/projected/19b6882c-5393-4adc-ae92-a45a12880e5c-kube-api-access-sf8hk\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.208233 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.208123 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-podinfo\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.208288 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.208263 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-workload-certs\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.208288 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.208270 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-data\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.208397 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.208301 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-credential-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.208397 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.208323 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-workload-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.208536 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.208516 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-workload-certs\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.208719 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.208693 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b6882c-5393-4adc-ae92-a45a12880e5c-istiod-ca-cert\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.210402 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.210374 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-envoy\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.210637 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.210621 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-podinfo\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.265127 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.265092 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-token\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.265240 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.265216 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf8hk\" (UniqueName: \"kubernetes.io/projected/19b6882c-5393-4adc-ae92-a45a12880e5c-kube-api-access-sf8hk\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.394251 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.394226 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:37.523806 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.523770 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:37.818443 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.818420 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6"] Nov 04 05:06:37.820440 ip-10-0-141-132 kubenswrapper[4581]: W1104 05:06:37.820412 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b6882c_5393_4adc_ae92_a45a12880e5c.slice/crio-47b8a4e19a45c91645b387b014f6e3749a609f5d23c8a38ed449aa3689886bf1 WatchSource:0}: Error finding container 47b8a4e19a45c91645b387b014f6e3749a609f5d23c8a38ed449aa3689886bf1: Status 404 returned error can't find the container with id 47b8a4e19a45c91645b387b014f6e3749a609f5d23c8a38ed449aa3689886bf1 Nov 04 05:06:37.822815 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.822779 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:06:37.822899 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.822850 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:06:37.822899 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:37.822884 4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 05:06:38.703913 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:38.703877 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" event={"ID":"19b6882c-5393-4adc-ae92-a45a12880e5c","Type":"ContainerStarted","Data":"7fdbcd042adb641953d28f77296ff45a5b8cf5e7f55ac805931c72384121d033"} Nov 04 05:06:38.704331 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:38.703935 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" event={"ID":"19b6882c-5393-4adc-ae92-a45a12880e5c","Type":"ContainerStarted","Data":"47b8a4e19a45c91645b387b014f6e3749a609f5d23c8a38ed449aa3689886bf1"} Nov 04 05:06:38.873599 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:38.873546 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" podStartSLOduration=1.873530597 podStartE2EDuration="1.873530597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:06:38.872208242 +0000 UTC m=+1157.763065262" watchObservedRunningTime="2026-11-04 05:06:38.873530597 +0000 UTC m=+1157.764387617" Nov 04 05:06:39.394629 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:39.394592 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:39.399503 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:39.399477 4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:39.710093 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:39.710023 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:39.710823 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:39.710804 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:42.388294 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:42.388256 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:42.936954 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:42.936914 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:42.937662 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:42.937642 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6"] Nov 04 05:06:42.937827 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:42.937810 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" podUID="b652ed62-2de7-4888-91e1-686213b028f3" containerName="istio-proxy" containerID="cri-o://d0bd0db28c6f21c1df6800eb9c2bfcb0be95a1bd66d521762f6a3d1f808911f8" gracePeriod=30 Nov 04 05:06:42.947519 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:42.947492 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d"] Nov 04 05:06:42.947725 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:42.947703 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" podUID="2b61c48c-bc70-4cc1-977f-494f8f637943" containerName="istio-proxy" containerID="cri-o://fb83921215a0f0c8505251d7f0f311f6bf809da333f5661cede46952a937a077" gracePeriod=30 Nov 04 05:06:42.972075 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:42.972045 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9"] Nov 04 05:06:42.972257 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:42.972238 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" podUID="37d77404-f335-45d1-8335-9a725c555979" containerName="istio-proxy" containerID="cri-o://0e7137c2769281f3885b0737fecd776c14749ffb0ed81fa4324f4d763b8c9495" gracePeriod=30 Nov 04 05:06:42.987781 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:42.987747 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6"] Nov 04 05:06:42.987927 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:42.987908 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" podUID="19b6882c-5393-4adc-ae92-a45a12880e5c" containerName="istio-proxy" containerID="cri-o://7fdbcd042adb641953d28f77296ff45a5b8cf5e7f55ac805931c72384121d033" gracePeriod=30 Nov 04 05:06:43.763772 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:43.763733 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:43.897929 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:43.897896 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:44.716037 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:44.715998 4581 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 05:06:44.716198 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:44.716066 4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" podUID="b652ed62-2de7-4888-91e1-686213b028f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 05:06:45.799141 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:45.799106 4581 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 04 05:06:45.799549 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:45.799174 4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" podUID="37d77404-f335-45d1-8335-9a725c555979" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 05:06:45.814895 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:45.814860 4581 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 05:06:45.815055 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:45.814913 4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" podUID="2b61c48c-bc70-4cc1-977f-494f8f637943" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 05:06:47.885823 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:06:47.885790 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 05:06:48.253440 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.253414 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:06:48.304444 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.304414 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:48.313287 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.313256 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:48.316448 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.316428 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:48.334841 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.334820 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-workload-certs\") pod \"b652ed62-2de7-4888-91e1-686213b028f3\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " Nov 04 05:06:48.334929 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.334915 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b652ed62-2de7-4888-91e1-686213b028f3-kube-api-access-kdwx7\" (UniqueName: \"kubernetes.io/projected/b652ed62-2de7-4888-91e1-686213b028f3-kube-api-access-kdwx7\") pod \"b652ed62-2de7-4888-91e1-686213b028f3\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " Nov 04 05:06:48.334977 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.334945 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-workload-socket\") pod \"b652ed62-2de7-4888-91e1-686213b028f3\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " Nov 04 05:06:48.334977 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.334968 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-istio-envoy\") pod \"b652ed62-2de7-4888-91e1-686213b028f3\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " Nov 04 05:06:48.335118 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.335027 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b652ed62-2de7-4888-91e1-686213b028f3-istio-token\" (UniqueName: \"kubernetes.io/projected/b652ed62-2de7-4888-91e1-686213b028f3-istio-token\") pod \"b652ed62-2de7-4888-91e1-686213b028f3\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " Nov 04 05:06:48.335118 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.335061 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-istio-data\") pod \"b652ed62-2de7-4888-91e1-686213b028f3\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " Nov 04 05:06:48.335118 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.335062 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-workload-certs" pod "b652ed62-2de7-4888-91e1-686213b028f3" (UID: "b652ed62-2de7-4888-91e1-686213b028f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.335118 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.335096 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b652ed62-2de7-4888-91e1-686213b028f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b652ed62-2de7-4888-91e1-686213b028f3-istiod-ca-cert\") pod \"b652ed62-2de7-4888-91e1-686213b028f3\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " Nov 04 05:06:48.335322 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.335186 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b652ed62-2de7-4888-91e1-686213b028f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b652ed62-2de7-4888-91e1-686213b028f3-istio-podinfo\") pod \"b652ed62-2de7-4888-91e1-686213b028f3\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " Nov 04 05:06:48.335322 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.335226 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-credential-socket\") pod \"b652ed62-2de7-4888-91e1-686213b028f3\" (UID: \"b652ed62-2de7-4888-91e1-686213b028f3\") " Nov 04 05:06:48.335322 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.335251 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-workload-socket" pod "b652ed62-2de7-4888-91e1-686213b028f3" (UID: "b652ed62-2de7-4888-91e1-686213b028f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.335486 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.335391 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-istio-data" pod "b652ed62-2de7-4888-91e1-686213b028f3" (UID: "b652ed62-2de7-4888-91e1-686213b028f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.335543 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.335503 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b652ed62-2de7-4888-91e1-686213b028f3-istiod-ca-cert" pod "b652ed62-2de7-4888-91e1-686213b028f3" (UID: "b652ed62-2de7-4888-91e1-686213b028f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:06:48.335610 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.335590 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-credential-socket" pod "b652ed62-2de7-4888-91e1-686213b028f3" (UID: "b652ed62-2de7-4888-91e1-686213b028f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.335941 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.335870 4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-workload-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.335941 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.335902 4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-istio-data\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.335941 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.335917 4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b652ed62-2de7-4888-91e1-686213b028f3-istiod-ca-cert\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.335941 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.335933 4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-credential-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.335941 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.335947 4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-workload-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.337434 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.337409 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b652ed62-2de7-4888-91e1-686213b028f3-istio-podinfo" pod "b652ed62-2de7-4888-91e1-686213b028f3" (UID: "b652ed62-2de7-4888-91e1-686213b028f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:06:48.337434 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.337414 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b652ed62-2de7-4888-91e1-686213b028f3-kube-api-access-kdwx7" pod "b652ed62-2de7-4888-91e1-686213b028f3" (UID: "b652ed62-2de7-4888-91e1-686213b028f3"). InnerVolumeSpecName "kube-api-access-kdwx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:48.337557 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.337434 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b652ed62-2de7-4888-91e1-686213b028f3-istio-token" pod "b652ed62-2de7-4888-91e1-686213b028f3" (UID: "b652ed62-2de7-4888-91e1-686213b028f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:48.337764 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.337748 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-istio-envoy" pod "b652ed62-2de7-4888-91e1-686213b028f3" (UID: "b652ed62-2de7-4888-91e1-686213b028f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.436320 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436243 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37d77404-f335-45d1-8335-9a725c555979-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d77404-f335-45d1-8335-9a725c555979-istiod-ca-cert\") pod \"37d77404-f335-45d1-8335-9a725c555979\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " Nov 04 05:06:48.436320 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436287 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-podinfo\") pod \"19b6882c-5393-4adc-ae92-a45a12880e5c\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " Nov 04 05:06:48.436320 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436309 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-data\") pod \"19b6882c-5393-4adc-ae92-a45a12880e5c\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " Nov 04 05:06:48.436594 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436336 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-istio-data\") pod \"37d77404-f335-45d1-8335-9a725c555979\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " Nov 04 05:06:48.436594 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436359 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-data\") pod \"2b61c48c-bc70-4cc1-977f-494f8f637943\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " Nov 04 05:06:48.436594 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436382 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-workload-certs\") pod \"2b61c48c-bc70-4cc1-977f-494f8f637943\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " Nov 04 05:06:48.436594 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436408 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-token\" (UniqueName: \"kubernetes.io/projected/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-token\") pod \"19b6882c-5393-4adc-ae92-a45a12880e5c\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " Nov 04 05:06:48.436594 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436451 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-credential-socket\") pod \"37d77404-f335-45d1-8335-9a725c555979\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " Nov 04 05:06:48.436594 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436497 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-envoy\") pod \"2b61c48c-bc70-4cc1-977f-494f8f637943\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " Nov 04 05:06:48.436594 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436543 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-workload-certs\") pod \"37d77404-f335-45d1-8335-9a725c555979\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " Nov 04 05:06:48.436594 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436591 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-envoy\") pod \"19b6882c-5393-4adc-ae92-a45a12880e5c\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " Nov 04 05:06:48.437026 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436631 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-workload-socket\") pod \"2b61c48c-bc70-4cc1-977f-494f8f637943\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " Nov 04 05:06:48.437026 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436658 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19b6882c-5393-4adc-ae92-a45a12880e5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b6882c-5393-4adc-ae92-a45a12880e5c-istiod-ca-cert\") pod \"19b6882c-5393-4adc-ae92-a45a12880e5c\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " Nov 04 05:06:48.437026 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436664 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-data" pod "2b61c48c-bc70-4cc1-977f-494f8f637943" (UID: "2b61c48c-bc70-4cc1-977f-494f8f637943"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.437026 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436685 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-workload-certs\") pod \"19b6882c-5393-4adc-ae92-a45a12880e5c\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " Nov 04 05:06:48.437026 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436682 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-workload-certs" pod "2b61c48c-bc70-4cc1-977f-494f8f637943" (UID: "2b61c48c-bc70-4cc1-977f-494f8f637943"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.437026 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436686 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-istio-data" pod "37d77404-f335-45d1-8335-9a725c555979" (UID: "37d77404-f335-45d1-8335-9a725c555979"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.437026 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436714 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d77404-f335-45d1-8335-9a725c555979-istiod-ca-cert" pod "37d77404-f335-45d1-8335-9a725c555979" (UID: "37d77404-f335-45d1-8335-9a725c555979"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:06:48.437026 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436725 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37d77404-f335-45d1-8335-9a725c555979-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d77404-f335-45d1-8335-9a725c555979-istio-podinfo\") pod \"37d77404-f335-45d1-8335-9a725c555979\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " Nov 04 05:06:48.437026 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436789 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-credential-socket\") pod \"2b61c48c-bc70-4cc1-977f-494f8f637943\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " Nov 04 05:06:48.437026 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436908 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-istio-envoy\") pod \"37d77404-f335-45d1-8335-9a725c555979\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " Nov 04 05:06:48.437026 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436915 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-workload-certs" pod "37d77404-f335-45d1-8335-9a725c555979" (UID: "37d77404-f335-45d1-8335-9a725c555979"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.437026 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436925 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-workload-socket" pod "2b61c48c-bc70-4cc1-977f-494f8f637943" (UID: "2b61c48c-bc70-4cc1-977f-494f8f637943"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.437026 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436941 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b61c48c-bc70-4cc1-977f-494f8f637943-kube-api-access-8wl4z\" (UniqueName: \"kubernetes.io/projected/2b61c48c-bc70-4cc1-977f-494f8f637943-kube-api-access-8wl4z\") pod \"2b61c48c-bc70-4cc1-977f-494f8f637943\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " Nov 04 05:06:48.437026 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436935 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-credential-socket" pod "37d77404-f335-45d1-8335-9a725c555979" (UID: "37d77404-f335-45d1-8335-9a725c555979"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.437026 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436973 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d77404-f335-45d1-8335-9a725c555979-kube-api-access-f75gj\" (UniqueName: \"kubernetes.io/projected/37d77404-f335-45d1-8335-9a725c555979-kube-api-access-f75gj\") pod \"37d77404-f335-45d1-8335-9a725c555979\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " Nov 04 05:06:48.437026 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.436980 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-data" pod "19b6882c-5393-4adc-ae92-a45a12880e5c" (UID: "19b6882c-5393-4adc-ae92-a45a12880e5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.438347 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.437083 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b61c48c-bc70-4cc1-977f-494f8f637943-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b61c48c-bc70-4cc1-977f-494f8f637943-istiod-ca-cert\") pod \"2b61c48c-bc70-4cc1-977f-494f8f637943\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " Nov 04 05:06:48.438347 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.437212 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-workload-certs" pod "19b6882c-5393-4adc-ae92-a45a12880e5c" (UID: "19b6882c-5393-4adc-ae92-a45a12880e5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.438347 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.437255 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b6882c-5393-4adc-ae92-a45a12880e5c-kube-api-access-sf8hk\" (UniqueName: \"kubernetes.io/projected/19b6882c-5393-4adc-ae92-a45a12880e5c-kube-api-access-sf8hk\") pod \"19b6882c-5393-4adc-ae92-a45a12880e5c\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " Nov 04 05:06:48.438347 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.437294 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b6882c-5393-4adc-ae92-a45a12880e5c-istiod-ca-cert" pod "19b6882c-5393-4adc-ae92-a45a12880e5c" (UID: "19b6882c-5393-4adc-ae92-a45a12880e5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:06:48.438347 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.437297 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-workload-socket\") pod \"19b6882c-5393-4adc-ae92-a45a12880e5c\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " Nov 04 05:06:48.438347 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.437349 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-token\" (UniqueName: \"kubernetes.io/projected/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-token\") pod \"2b61c48c-bc70-4cc1-977f-494f8f637943\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " Nov 04 05:06:48.438347 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.437379 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-podinfo\") pod \"2b61c48c-bc70-4cc1-977f-494f8f637943\" (UID: \"2b61c48c-bc70-4cc1-977f-494f8f637943\") " Nov 04 05:06:48.438347 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.437413 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-credential-socket\") pod \"19b6882c-5393-4adc-ae92-a45a12880e5c\" (UID: \"19b6882c-5393-4adc-ae92-a45a12880e5c\") " Nov 04 05:06:48.438347 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.437512 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-workload-socket" pod "19b6882c-5393-4adc-ae92-a45a12880e5c" (UID: "19b6882c-5393-4adc-ae92-a45a12880e5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.438347 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.437531 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-workload-socket\") pod \"37d77404-f335-45d1-8335-9a725c555979\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " Nov 04 05:06:48.438347 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.437573 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d77404-f335-45d1-8335-9a725c555979-istio-token\" (UniqueName: \"kubernetes.io/projected/37d77404-f335-45d1-8335-9a725c555979-istio-token\") pod \"37d77404-f335-45d1-8335-9a725c555979\" (UID: \"37d77404-f335-45d1-8335-9a725c555979\") " Nov 04 05:06:48.438347 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.438202 4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-workload-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.438347 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.438225 4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b6882c-5393-4adc-ae92-a45a12880e5c-istiod-ca-cert\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.438347 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.438240 4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-workload-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.438347 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.438256 4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b652ed62-2de7-4888-91e1-686213b028f3-istio-podinfo\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.438347 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.438271 4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-workload-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.438347 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.438286 4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d77404-f335-45d1-8335-9a725c555979-istiod-ca-cert\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.438347 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.438301 4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-data\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.438347 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.438315 4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-istio-data\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.438347 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.438329 4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-data\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.438347 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.438343 4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-workload-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.438347 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.438357 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdwx7\" (UniqueName: \"kubernetes.io/projected/b652ed62-2de7-4888-91e1-686213b028f3-kube-api-access-kdwx7\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.439732 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.438372 4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-credential-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.439732 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.438387 4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b652ed62-2de7-4888-91e1-686213b028f3-istio-envoy\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.439732 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.438401 4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b652ed62-2de7-4888-91e1-686213b028f3-istio-token\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.439732 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.438415 4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-workload-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.439732 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.439447 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-podinfo" pod "19b6882c-5393-4adc-ae92-a45a12880e5c" (UID: "19b6882c-5393-4adc-ae92-a45a12880e5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:06:48.439732 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.439620 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37d77404-f335-45d1-8335-9a725c555979-istio-podinfo" pod "37d77404-f335-45d1-8335-9a725c555979" (UID: "37d77404-f335-45d1-8335-9a725c555979"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:06:48.439732 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.439623 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-envoy" pod "19b6882c-5393-4adc-ae92-a45a12880e5c" (UID: "19b6882c-5393-4adc-ae92-a45a12880e5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.440134 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.439895 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-envoy" pod "2b61c48c-bc70-4cc1-977f-494f8f637943" (UID: "2b61c48c-bc70-4cc1-977f-494f8f637943"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.440134 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.439937 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b61c48c-bc70-4cc1-977f-494f8f637943-istiod-ca-cert" pod "2b61c48c-bc70-4cc1-977f-494f8f637943" (UID: "2b61c48c-bc70-4cc1-977f-494f8f637943"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:06:48.440134 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.440074 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b61c48c-bc70-4cc1-977f-494f8f637943-kube-api-access-8wl4z" pod "2b61c48c-bc70-4cc1-977f-494f8f637943" (UID: "2b61c48c-bc70-4cc1-977f-494f8f637943"). InnerVolumeSpecName "kube-api-access-8wl4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:48.440294 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.440219 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-token" pod "19b6882c-5393-4adc-ae92-a45a12880e5c" (UID: "19b6882c-5393-4adc-ae92-a45a12880e5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:48.440294 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.440240 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-credential-socket" pod "2b61c48c-bc70-4cc1-977f-494f8f637943" (UID: "2b61c48c-bc70-4cc1-977f-494f8f637943"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.440499 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.440476 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-credential-socket" pod "19b6882c-5393-4adc-ae92-a45a12880e5c" (UID: "19b6882c-5393-4adc-ae92-a45a12880e5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.440499 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.440485 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-workload-socket" pod "37d77404-f335-45d1-8335-9a725c555979" (UID: "37d77404-f335-45d1-8335-9a725c555979"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.441085 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.441049 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d77404-f335-45d1-8335-9a725c555979-istio-token" pod "37d77404-f335-45d1-8335-9a725c555979" (UID: "37d77404-f335-45d1-8335-9a725c555979"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:48.441208 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.441130 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-istio-envoy" pod "37d77404-f335-45d1-8335-9a725c555979" (UID: "37d77404-f335-45d1-8335-9a725c555979"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.442089 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.442068 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d77404-f335-45d1-8335-9a725c555979-kube-api-access-f75gj" pod "37d77404-f335-45d1-8335-9a725c555979" (UID: "37d77404-f335-45d1-8335-9a725c555979"). InnerVolumeSpecName "kube-api-access-f75gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:48.442208 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.442187 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-podinfo" pod "2b61c48c-bc70-4cc1-977f-494f8f637943" (UID: "2b61c48c-bc70-4cc1-977f-494f8f637943"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:06:48.442306 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.442292 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-token" pod "2b61c48c-bc70-4cc1-977f-494f8f637943" (UID: "2b61c48c-bc70-4cc1-977f-494f8f637943"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:48.442474 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.442462 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b6882c-5393-4adc-ae92-a45a12880e5c-kube-api-access-sf8hk" pod "19b6882c-5393-4adc-ae92-a45a12880e5c" (UID: "19b6882c-5393-4adc-ae92-a45a12880e5c"). InnerVolumeSpecName "kube-api-access-sf8hk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:48.539862 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.539827 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf8hk\" (UniqueName: \"kubernetes.io/projected/19b6882c-5393-4adc-ae92-a45a12880e5c-kube-api-access-sf8hk\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.540070 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.539884 4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-token\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.540070 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.539902 4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-podinfo\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.540070 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.539916 4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-credential-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.540070 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.539929 4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-workload-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.540070 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.539943 4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d77404-f335-45d1-8335-9a725c555979-istio-token\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.540070 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.539956 4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-podinfo\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.540070 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.539970 4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-token\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.540070 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.540002 4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-istio-envoy\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.540070 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.540016 4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b6882c-5393-4adc-ae92-a45a12880e5c-istio-envoy\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.540070 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.540028 4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d77404-f335-45d1-8335-9a725c555979-istio-podinfo\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.540070 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.540040 4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b61c48c-bc70-4cc1-977f-494f8f637943-credential-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.540070 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.540052 4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d77404-f335-45d1-8335-9a725c555979-istio-envoy\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.540070 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.540065 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wl4z\" (UniqueName: \"kubernetes.io/projected/2b61c48c-bc70-4cc1-977f-494f8f637943-kube-api-access-8wl4z\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.540070 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.540078 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f75gj\" (UniqueName: \"kubernetes.io/projected/37d77404-f335-45d1-8335-9a725c555979-kube-api-access-f75gj\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.540658 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.540092 4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b61c48c-bc70-4cc1-977f-494f8f637943-istiod-ca-cert\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.759287 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.759197 4581 generic.go:356] "Generic (PLEG): container finished" podID="37d77404-f335-45d1-8335-9a725c555979" containerID="0e7137c2769281f3885b0737fecd776c14749ffb0ed81fa4324f4d763b8c9495" exitCode=0 Nov 04 05:06:48.759287 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.759265 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" Nov 04 05:06:48.759287 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.759273 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" event={"ID":"37d77404-f335-45d1-8335-9a725c555979","Type":"ContainerDied","Data":"0e7137c2769281f3885b0737fecd776c14749ffb0ed81fa4324f4d763b8c9495"} Nov 04 05:06:48.759520 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.759359 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9" event={"ID":"37d77404-f335-45d1-8335-9a725c555979","Type":"ContainerDied","Data":"da8a571f7c1f8295a66067289ee526395792c04c3dfff8be048e274cdd0e8869"} Nov 04 05:06:48.759520 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.759388 4581 scope.go:122] "RemoveContainer" containerID="0e7137c2769281f3885b0737fecd776c14749ffb0ed81fa4324f4d763b8c9495" Nov 04 05:06:48.760621 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.760598 4581 generic.go:356] "Generic (PLEG): container finished" podID="19b6882c-5393-4adc-ae92-a45a12880e5c" containerID="7fdbcd042adb641953d28f77296ff45a5b8cf5e7f55ac805931c72384121d033" exitCode=0 Nov 04 05:06:48.760765 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.760753 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" Nov 04 05:06:48.762334 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.762077 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" event={"ID":"19b6882c-5393-4adc-ae92-a45a12880e5c","Type":"ContainerDied","Data":"7fdbcd042adb641953d28f77296ff45a5b8cf5e7f55ac805931c72384121d033"} Nov 04 05:06:48.762334 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.762173 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6" event={"ID":"19b6882c-5393-4adc-ae92-a45a12880e5c","Type":"ContainerDied","Data":"47b8a4e19a45c91645b387b014f6e3749a609f5d23c8a38ed449aa3689886bf1"} Nov 04 05:06:48.763912 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.763891 4581 generic.go:356] "Generic (PLEG): container finished" podID="b652ed62-2de7-4888-91e1-686213b028f3" containerID="d0bd0db28c6f21c1df6800eb9c2bfcb0be95a1bd66d521762f6a3d1f808911f8" exitCode=0 Nov 04 05:06:48.764057 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.764033 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" event={"ID":"b652ed62-2de7-4888-91e1-686213b028f3","Type":"ContainerDied","Data":"d0bd0db28c6f21c1df6800eb9c2bfcb0be95a1bd66d521762f6a3d1f808911f8"} Nov 04 05:06:48.764135 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.764086 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" event={"ID":"b652ed62-2de7-4888-91e1-686213b028f3","Type":"ContainerDied","Data":"db088cce490240018f1d12aecf6349a2bcf52c108f35eb2785ca709d56b4cc12"} Nov 04 05:06:48.764135 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.764041 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6" Nov 04 05:06:48.765698 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.765675 4581 generic.go:356] "Generic (PLEG): container finished" podID="2b61c48c-bc70-4cc1-977f-494f8f637943" containerID="fb83921215a0f0c8505251d7f0f311f6bf809da333f5661cede46952a937a077" exitCode=0 Nov 04 05:06:48.765823 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.765738 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" event={"ID":"2b61c48c-bc70-4cc1-977f-494f8f637943","Type":"ContainerDied","Data":"fb83921215a0f0c8505251d7f0f311f6bf809da333f5661cede46952a937a077"} Nov 04 05:06:48.765823 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.765776 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" event={"ID":"2b61c48c-bc70-4cc1-977f-494f8f637943","Type":"ContainerDied","Data":"d9120127057c13863676e6d92be2a6ee0995874cbcb6b1f675a3ab6a9aab7745"} Nov 04 05:06:48.765823 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.765738 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d" Nov 04 05:06:48.771662 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.771644 4581 scope.go:122] "RemoveContainer" containerID="0e7137c2769281f3885b0737fecd776c14749ffb0ed81fa4324f4d763b8c9495" Nov 04 05:06:48.771941 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:06:48.771914 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e7137c2769281f3885b0737fecd776c14749ffb0ed81fa4324f4d763b8c9495\": container with ID starting with 0e7137c2769281f3885b0737fecd776c14749ffb0ed81fa4324f4d763b8c9495 not found: ID does not exist" containerID="0e7137c2769281f3885b0737fecd776c14749ffb0ed81fa4324f4d763b8c9495" Nov 04 05:06:48.772042 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.771951 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e7137c2769281f3885b0737fecd776c14749ffb0ed81fa4324f4d763b8c9495"} err="failed to get container status \"0e7137c2769281f3885b0737fecd776c14749ffb0ed81fa4324f4d763b8c9495\": rpc error: code = NotFound desc = could not find container \"0e7137c2769281f3885b0737fecd776c14749ffb0ed81fa4324f4d763b8c9495\": container with ID starting with 0e7137c2769281f3885b0737fecd776c14749ffb0ed81fa4324f4d763b8c9495 not found: ID does not exist" Nov 04 05:06:48.772042 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.771973 4581 scope.go:122] "RemoveContainer" containerID="7fdbcd042adb641953d28f77296ff45a5b8cf5e7f55ac805931c72384121d033" Nov 04 05:06:48.786637 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.786615 4581 scope.go:122] "RemoveContainer" containerID="7fdbcd042adb641953d28f77296ff45a5b8cf5e7f55ac805931c72384121d033" Nov 04 05:06:48.787058 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:06:48.787027 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fdbcd042adb641953d28f77296ff45a5b8cf5e7f55ac805931c72384121d033\": container with ID starting with 7fdbcd042adb641953d28f77296ff45a5b8cf5e7f55ac805931c72384121d033 not found: ID does not exist" containerID="7fdbcd042adb641953d28f77296ff45a5b8cf5e7f55ac805931c72384121d033" Nov 04 05:06:48.787171 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.787069 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fdbcd042adb641953d28f77296ff45a5b8cf5e7f55ac805931c72384121d033"} err="failed to get container status \"7fdbcd042adb641953d28f77296ff45a5b8cf5e7f55ac805931c72384121d033\": rpc error: code = NotFound desc = could not find container \"7fdbcd042adb641953d28f77296ff45a5b8cf5e7f55ac805931c72384121d033\": container with ID starting with 7fdbcd042adb641953d28f77296ff45a5b8cf5e7f55ac805931c72384121d033 not found: ID does not exist" Nov 04 05:06:48.787171 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.787093 4581 scope.go:122] "RemoveContainer" containerID="d0bd0db28c6f21c1df6800eb9c2bfcb0be95a1bd66d521762f6a3d1f808911f8" Nov 04 05:06:48.797314 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.797296 4581 scope.go:122] "RemoveContainer" containerID="d0bd0db28c6f21c1df6800eb9c2bfcb0be95a1bd66d521762f6a3d1f808911f8" Nov 04 05:06:48.797599 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:06:48.797579 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0bd0db28c6f21c1df6800eb9c2bfcb0be95a1bd66d521762f6a3d1f808911f8\": container with ID starting with d0bd0db28c6f21c1df6800eb9c2bfcb0be95a1bd66d521762f6a3d1f808911f8 not found: ID does not exist" containerID="d0bd0db28c6f21c1df6800eb9c2bfcb0be95a1bd66d521762f6a3d1f808911f8" Nov 04 05:06:48.797718 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.797605 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0bd0db28c6f21c1df6800eb9c2bfcb0be95a1bd66d521762f6a3d1f808911f8"} err="failed to get container status \"d0bd0db28c6f21c1df6800eb9c2bfcb0be95a1bd66d521762f6a3d1f808911f8\": rpc error: code = NotFound desc = could not find container \"d0bd0db28c6f21c1df6800eb9c2bfcb0be95a1bd66d521762f6a3d1f808911f8\": container with ID starting with d0bd0db28c6f21c1df6800eb9c2bfcb0be95a1bd66d521762f6a3d1f808911f8 not found: ID does not exist" Nov 04 05:06:48.797718 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.797627 4581 scope.go:122] "RemoveContainer" containerID="fb83921215a0f0c8505251d7f0f311f6bf809da333f5661cede46952a937a077" Nov 04 05:06:48.805753 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.805733 4581 scope.go:122] "RemoveContainer" containerID="fb83921215a0f0c8505251d7f0f311f6bf809da333f5661cede46952a937a077" Nov 04 05:06:48.806055 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:06:48.806032 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb83921215a0f0c8505251d7f0f311f6bf809da333f5661cede46952a937a077\": container with ID starting with fb83921215a0f0c8505251d7f0f311f6bf809da333f5661cede46952a937a077 not found: ID does not exist" containerID="fb83921215a0f0c8505251d7f0f311f6bf809da333f5661cede46952a937a077" Nov 04 05:06:48.806160 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:48.806064 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb83921215a0f0c8505251d7f0f311f6bf809da333f5661cede46952a937a077"} err="failed to get container status \"fb83921215a0f0c8505251d7f0f311f6bf809da333f5661cede46952a937a077\": rpc error: code = NotFound desc = could not find container \"fb83921215a0f0c8505251d7f0f311f6bf809da333f5661cede46952a937a077\": container with ID starting with fb83921215a0f0c8505251d7f0f311f6bf809da333f5661cede46952a937a077 not found: ID does not exist" Nov 04 05:06:49.052324 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:49.052246 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9"] Nov 04 05:06:49.057705 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:49.057679 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9"] Nov 04 05:06:49.169067 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:49.169031 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d"] Nov 04 05:06:49.175332 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:49.175305 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d"] Nov 04 05:06:49.207529 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:49.207501 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:49.229293 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:49.229270 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b61c48c-bc70-4cc1-977f-494f8f637943" path="/var/lib/kubelet/pods/2b61c48c-bc70-4cc1-977f-494f8f637943/volumes" Nov 04 05:06:49.229737 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:49.229724 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37d77404-f335-45d1-8335-9a725c555979" path="/var/lib/kubelet/pods/37d77404-f335-45d1-8335-9a725c555979/volumes" Nov 04 05:06:49.285407 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:49.285374 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6"] Nov 04 05:06:49.289242 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:49.289220 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6"] Nov 04 05:06:49.349810 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:49.349726 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:49.399832 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:49.399800 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6"] Nov 04 05:06:49.406857 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:49.406834 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6"] Nov 04 05:06:50.143543 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:50.143512 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:06:51.229594 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:51.229564 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19b6882c-5393-4adc-ae92-a45a12880e5c" path="/var/lib/kubelet/pods/19b6882c-5393-4adc-ae92-a45a12880e5c/volumes" Nov 04 05:06:51.230050 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:06:51.230036 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b652ed62-2de7-4888-91e1-686213b028f3" path="/var/lib/kubelet/pods/b652ed62-2de7-4888-91e1-686213b028f3/volumes" Nov 04 05:07:02.804937 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:02.804906 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:07:17.885869 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:07:17.885819 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 05:07:19.294087 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:19.294055 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-qprw2_fa715875-4789-4276-8b9e-c809de6606b6/manager/0.log" Nov 04 05:07:19.566203 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:19.566128 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc"] Nov 04 05:07:19.566393 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:19.566372 4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" podUID="a91d185a-8a3f-4a08-9395-72e9a92265c8" containerName="istio-proxy" containerID="cri-o://9a3c03eda84cdce5dff28957661ad15e6de30431da0dddcb178d9c22878b3bff" gracePeriod=30 Nov 04 05:07:21.474470 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:21.474436 4581 scope.go:122] "RemoveContainer" containerID="18504455d82ab778be4b6847592cac5c263006b128509002f016a5a8a4676d4d" Nov 04 05:07:24.820453 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.820427 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 05:07:24.876317 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.876274 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-workload-certs\") pod \"a91d185a-8a3f-4a08-9395-72e9a92265c8\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " Nov 04 05:07:24.876485 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.876365 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-podinfo\") pod \"a91d185a-8a3f-4a08-9395-72e9a92265c8\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " Nov 04 05:07:24.876485 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.876395 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-envoy\") pod \"a91d185a-8a3f-4a08-9395-72e9a92265c8\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " Nov 04 05:07:24.876485 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.876463 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-data\") pod \"a91d185a-8a3f-4a08-9395-72e9a92265c8\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " Nov 04 05:07:24.876608 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.876522 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-credential-socket\") pod \"a91d185a-8a3f-4a08-9395-72e9a92265c8\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " Nov 04 05:07:24.876608 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.876566 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-token\" (UniqueName: \"kubernetes.io/projected/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-token\") pod \"a91d185a-8a3f-4a08-9395-72e9a92265c8\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " Nov 04 05:07:24.876608 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.876587 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-workload-certs" pod "a91d185a-8a3f-4a08-9395-72e9a92265c8" (UID: "a91d185a-8a3f-4a08-9395-72e9a92265c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:07:24.876608 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.876596 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a91d185a-8a3f-4a08-9395-72e9a92265c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a91d185a-8a3f-4a08-9395-72e9a92265c8-istiod-ca-cert\") pod \"a91d185a-8a3f-4a08-9395-72e9a92265c8\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " Nov 04 05:07:24.876813 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.876622 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a91d185a-8a3f-4a08-9395-72e9a92265c8-kube-api-access-mwdsx\" (UniqueName: \"kubernetes.io/projected/a91d185a-8a3f-4a08-9395-72e9a92265c8-kube-api-access-mwdsx\") pod \"a91d185a-8a3f-4a08-9395-72e9a92265c8\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " Nov 04 05:07:24.876813 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.876698 4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-workload-socket\") pod \"a91d185a-8a3f-4a08-9395-72e9a92265c8\" (UID: \"a91d185a-8a3f-4a08-9395-72e9a92265c8\") " Nov 04 05:07:24.876813 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.876785 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-data" pod "a91d185a-8a3f-4a08-9395-72e9a92265c8" (UID: "a91d185a-8a3f-4a08-9395-72e9a92265c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:07:24.876966 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.876898 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-credential-socket" pod "a91d185a-8a3f-4a08-9395-72e9a92265c8" (UID: "a91d185a-8a3f-4a08-9395-72e9a92265c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:07:24.877064 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.877024 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-workload-socket" pod "a91d185a-8a3f-4a08-9395-72e9a92265c8" (UID: "a91d185a-8a3f-4a08-9395-72e9a92265c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:07:24.877124 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.877058 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a91d185a-8a3f-4a08-9395-72e9a92265c8-istiod-ca-cert" pod "a91d185a-8a3f-4a08-9395-72e9a92265c8" (UID: "a91d185a-8a3f-4a08-9395-72e9a92265c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:07:24.877194 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.877176 4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-credential-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:07:24.877234 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.877201 4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a91d185a-8a3f-4a08-9395-72e9a92265c8-istiod-ca-cert\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:07:24.877234 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.877215 4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-workload-socket\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:07:24.877234 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.877229 4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-workload-certs\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:07:24.877328 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.877243 4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-data\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:07:24.878867 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.878840 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-podinfo" pod "a91d185a-8a3f-4a08-9395-72e9a92265c8" (UID: "a91d185a-8a3f-4a08-9395-72e9a92265c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:07:24.878976 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.878888 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-envoy" pod "a91d185a-8a3f-4a08-9395-72e9a92265c8" (UID: "a91d185a-8a3f-4a08-9395-72e9a92265c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:07:24.878976 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.878907 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a91d185a-8a3f-4a08-9395-72e9a92265c8-kube-api-access-mwdsx" pod "a91d185a-8a3f-4a08-9395-72e9a92265c8" (UID: "a91d185a-8a3f-4a08-9395-72e9a92265c8"). InnerVolumeSpecName "kube-api-access-mwdsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:07:24.879199 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.879183 4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-token" pod "a91d185a-8a3f-4a08-9395-72e9a92265c8" (UID: "a91d185a-8a3f-4a08-9395-72e9a92265c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:07:24.956532 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.956499 4581 generic.go:356] "Generic (PLEG): container finished" podID="a91d185a-8a3f-4a08-9395-72e9a92265c8" containerID="9a3c03eda84cdce5dff28957661ad15e6de30431da0dddcb178d9c22878b3bff" exitCode=0 Nov 04 05:07:24.956725 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.956542 4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" Nov 04 05:07:24.956725 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.956583 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" event={"ID":"a91d185a-8a3f-4a08-9395-72e9a92265c8","Type":"ContainerDied","Data":"9a3c03eda84cdce5dff28957661ad15e6de30431da0dddcb178d9c22878b3bff"} Nov 04 05:07:24.956725 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.956643 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc" event={"ID":"a91d185a-8a3f-4a08-9395-72e9a92265c8","Type":"ContainerDied","Data":"57d46108f35865374dc571ec6f4a7523da33952e78f816e2a0a223294139b327"} Nov 04 05:07:24.956725 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.956674 4581 scope.go:122] "RemoveContainer" containerID="9a3c03eda84cdce5dff28957661ad15e6de30431da0dddcb178d9c22878b3bff" Nov 04 05:07:24.971401 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.971382 4581 scope.go:122] "RemoveContainer" containerID="9a3c03eda84cdce5dff28957661ad15e6de30431da0dddcb178d9c22878b3bff" Nov 04 05:07:24.971687 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:07:24.971665 4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a3c03eda84cdce5dff28957661ad15e6de30431da0dddcb178d9c22878b3bff\": container with ID starting with 9a3c03eda84cdce5dff28957661ad15e6de30431da0dddcb178d9c22878b3bff not found: ID does not exist" containerID="9a3c03eda84cdce5dff28957661ad15e6de30431da0dddcb178d9c22878b3bff" Nov 04 05:07:24.971762 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.971697 4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a3c03eda84cdce5dff28957661ad15e6de30431da0dddcb178d9c22878b3bff"} err="failed to get container status \"9a3c03eda84cdce5dff28957661ad15e6de30431da0dddcb178d9c22878b3bff\": rpc error: code = NotFound desc = could not find container \"9a3c03eda84cdce5dff28957661ad15e6de30431da0dddcb178d9c22878b3bff\": container with ID starting with 9a3c03eda84cdce5dff28957661ad15e6de30431da0dddcb178d9c22878b3bff not found: ID does not exist" Nov 04 05:07:24.978215 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.978184 4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-token\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:07:24.978215 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.978215 4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwdsx\" (UniqueName: \"kubernetes.io/projected/a91d185a-8a3f-4a08-9395-72e9a92265c8-kube-api-access-mwdsx\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:07:24.978348 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.978229 4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-podinfo\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:07:24.978348 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:24.978242 4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a91d185a-8a3f-4a08-9395-72e9a92265c8-istio-envoy\") on node \"ip-10-0-141-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:07:25.099835 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:25.099800 4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc"] Nov 04 05:07:25.104146 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:25.104120 4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc"] Nov 04 05:07:25.229834 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:25.229763 4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a91d185a-8a3f-4a08-9395-72e9a92265c8" path="/var/lib/kubelet/pods/a91d185a-8a3f-4a08-9395-72e9a92265c8/volumes" Nov 04 05:07:31.048778 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:31.048750 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7f4cf7d6f8-jdnlc_4b8f86bd-05d2-468a-bdfb-3558700e813c/maas-api/0.log" Nov 04 05:07:47.737674 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:07:47.737602 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-4bhfh_a19b0538-b992-48a4-8566-960d59838722/kuadrant-console-plugin/0.log" Nov 04 05:07:47.884715 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:07:47.884682 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 05:08:00.541220 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:08:00.541175 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-546d78d5c-g8s92_63a561a1-32d4-41b1-a285-4e82379ac8d4/kube-auth-proxy/0.log" Nov 04 05:08:01.980161 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:08:01.980125 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-xvj6h_a97545a8-870a-4ce9-907f-069b73552821/payload-processing/0.log" Nov 04 05:08:03.142597 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:08:03.142549 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:08:03.571332 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:08:03.571251 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-v454h_ae7aa7c9-fba5-4b19-9163-32de4ad7b567/main/0.log" Nov 04 05:08:04.313564 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:08:04.313534 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-mmz8r_1c20fd01-1367-4eef-be7c-acb7a33be306/main/0.log" Nov 04 05:08:04.596829 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:08:04.596738 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f_ca9c82d3-d38a-4910-8a3b-90c4412b508b/main/0.log" Nov 04 05:08:04.884847 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:08:04.884819 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-kp5cw_26f5c90c-d608-4b86-9909-5c1bf8fdbd24/main/0.log" Nov 04 05:08:17.886688 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:08:17.886647 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 05:08:18.204415 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:08:18.204345 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-qprw2_fa715875-4789-4276-8b9e-c809de6606b6/manager/0.log" Nov 04 05:08:19.291201 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:08:19.291173 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7f4cf7d6f8-jdnlc_4b8f86bd-05d2-468a-bdfb-3558700e813c/maas-api/0.log" Nov 04 05:08:25.380830 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:08:25.380803 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-4bhfh_a19b0538-b992-48a4-8566-960d59838722/kuadrant-console-plugin/0.log" Nov 04 05:08:37.620269 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:08:37.620236 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-546d78d5c-g8s92_63a561a1-32d4-41b1-a285-4e82379ac8d4/kube-auth-proxy/0.log" Nov 04 05:08:39.161855 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:08:39.161821 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-xvj6h_a97545a8-870a-4ce9-907f-069b73552821/payload-processing/0.log" Nov 04 05:08:39.991914 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:08:39.991883 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-v454h_ae7aa7c9-fba5-4b19-9163-32de4ad7b567/main/0.log" Nov 04 05:08:40.779747 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:08:40.779714 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-mmz8r_1c20fd01-1367-4eef-be7c-acb7a33be306/main/0.log" Nov 04 05:08:41.060022 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:08:41.059923 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svp6f_ca9c82d3-d38a-4910-8a3b-90c4412b508b/main/0.log" Nov 04 05:08:41.343126 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:08:41.343052 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-kp5cw_26f5c90c-d608-4b86-9909-5c1bf8fdbd24/main/0.log" Nov 04 05:08:47.885432 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:08:47.885392 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 05:09:03.476375 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:09:03.476274 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:09:17.886172 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:09:17.886135 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 05:09:47.885201 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:09:47.885165 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 05:09:49.906413 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:09:49.906375 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-xvj6h_a97545a8-870a-4ce9-907f-069b73552821/payload-processing/0.log" Nov 04 05:10:00.140146 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:00.140112 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-xvj6h_a97545a8-870a-4ce9-907f-069b73552821/payload-processing/0.log" Nov 04 05:10:03.814839 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:03.814798 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:10:17.885419 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:10:17.885381 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 05:10:30.354569 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:30.354521 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:10:30.505510 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:30.505474 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:10:47.889434 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:10:47.889348 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 05:10:56.388768 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:56.388734 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-xvj6h_a97545a8-870a-4ce9-907f-069b73552821/payload-processing/0.log" Nov 04 05:10:56.627652 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:56.627621 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-xvj6h_a97545a8-870a-4ce9-907f-069b73552821/payload-processing/0.log" Nov 04 05:10:56.997463 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:56.997422 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:10:57.015221 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.015184 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zx9qc/must-gather-wv8bd"] Nov 04 05:10:57.015632 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.015614 4581 cpu_manager.go:414] "containerMap: removing container" podUID="2b61c48c-bc70-4cc1-977f-494f8f637943" containerName="istio-proxy" Nov 04 05:10:57.015717 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.015634 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b61c48c-bc70-4cc1-977f-494f8f637943" containerName="istio-proxy" Nov 04 05:10:57.015717 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.015659 4581 cpu_manager.go:414] "containerMap: removing container" podUID="37d77404-f335-45d1-8335-9a725c555979" containerName="istio-proxy" Nov 04 05:10:57.015717 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.015668 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37d77404-f335-45d1-8335-9a725c555979" containerName="istio-proxy" Nov 04 05:10:57.015717 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.015701 4581 cpu_manager.go:414] "containerMap: removing container" podUID="b652ed62-2de7-4888-91e1-686213b028f3" containerName="istio-proxy" Nov 04 05:10:57.015717 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.015707 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b652ed62-2de7-4888-91e1-686213b028f3" containerName="istio-proxy" Nov 04 05:10:57.015967 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.015736 4581 cpu_manager.go:414] "containerMap: removing container" podUID="a91d185a-8a3f-4a08-9395-72e9a92265c8" containerName="istio-proxy" Nov 04 05:10:57.015967 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.015744 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a91d185a-8a3f-4a08-9395-72e9a92265c8" containerName="istio-proxy" Nov 04 05:10:57.015967 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.015754 4581 cpu_manager.go:414] "containerMap: removing container" podUID="19b6882c-5393-4adc-ae92-a45a12880e5c" containerName="istio-proxy" Nov 04 05:10:57.015967 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.015761 4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b6882c-5393-4adc-ae92-a45a12880e5c" containerName="istio-proxy" Nov 04 05:10:57.015967 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.015853 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b61c48c-bc70-4cc1-977f-494f8f637943" containerName="istio-proxy" Nov 04 05:10:57.015967 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.015868 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="37d77404-f335-45d1-8335-9a725c555979" containerName="istio-proxy" Nov 04 05:10:57.015967 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.015882 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="b652ed62-2de7-4888-91e1-686213b028f3" containerName="istio-proxy" Nov 04 05:10:57.015967 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.015895 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="a91d185a-8a3f-4a08-9395-72e9a92265c8" containerName="istio-proxy" Nov 04 05:10:57.015967 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.015906 4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b6882c-5393-4adc-ae92-a45a12880e5c" containerName="istio-proxy" Nov 04 05:10:57.019160 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.019140 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zx9qc/must-gather-wv8bd" Nov 04 05:10:57.072378 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.072353 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zx9qc\"/\"openshift-service-ca.crt\"" Nov 04 05:10:57.122467 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.122434 4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zx9qc\"/\"kube-root-ca.crt\"" Nov 04 05:10:57.122690 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.122673 4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zx9qc\"/\"default-dockercfg-qh69v\"" Nov 04 05:10:57.124552 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.124533 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zx9qc/must-gather-wv8bd"] Nov 04 05:10:57.140743 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.140704 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-996zc\" (UniqueName: \"kubernetes.io/projected/23567e72-6011-4373-9ce8-3ffe739682f2-kube-api-access-996zc\") pod \"must-gather-wv8bd\" (UID: \"23567e72-6011-4373-9ce8-3ffe739682f2\") " pod="openshift-must-gather-zx9qc/must-gather-wv8bd" Nov 04 05:10:57.140743 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.140749 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/23567e72-6011-4373-9ce8-3ffe739682f2-must-gather-output\") pod \"must-gather-wv8bd\" (UID: \"23567e72-6011-4373-9ce8-3ffe739682f2\") " pod="openshift-must-gather-zx9qc/must-gather-wv8bd" Nov 04 05:10:57.170452 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.170411 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:10:57.241659 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.241632 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-996zc\" (UniqueName: \"kubernetes.io/projected/23567e72-6011-4373-9ce8-3ffe739682f2-kube-api-access-996zc\") pod \"must-gather-wv8bd\" (UID: \"23567e72-6011-4373-9ce8-3ffe739682f2\") " pod="openshift-must-gather-zx9qc/must-gather-wv8bd" Nov 04 05:10:57.241829 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.241667 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/23567e72-6011-4373-9ce8-3ffe739682f2-must-gather-output\") pod \"must-gather-wv8bd\" (UID: \"23567e72-6011-4373-9ce8-3ffe739682f2\") " pod="openshift-must-gather-zx9qc/must-gather-wv8bd" Nov 04 05:10:57.242111 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.242093 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/23567e72-6011-4373-9ce8-3ffe739682f2-must-gather-output\") pod \"must-gather-wv8bd\" (UID: \"23567e72-6011-4373-9ce8-3ffe739682f2\") " pod="openshift-must-gather-zx9qc/must-gather-wv8bd" Nov 04 05:10:57.298927 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.298856 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-996zc\" (UniqueName: \"kubernetes.io/projected/23567e72-6011-4373-9ce8-3ffe739682f2-kube-api-access-996zc\") pod \"must-gather-wv8bd\" (UID: \"23567e72-6011-4373-9ce8-3ffe739682f2\") " pod="openshift-must-gather-zx9qc/must-gather-wv8bd" Nov 04 05:10:57.332853 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.332828 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zx9qc/must-gather-wv8bd" Nov 04 05:10:57.709233 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.709061 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zx9qc/must-gather-wv8bd"] Nov 04 05:10:57.711556 ip-10-0-141-132 kubenswrapper[4581]: W1104 05:10:57.711518 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23567e72_6011_4373_9ce8_3ffe739682f2.slice/crio-aaefc27a99b6bf72bffee419f6ba9757c9aad41e0dabf8005ab04b74c313971b WatchSource:0}: Error finding container aaefc27a99b6bf72bffee419f6ba9757c9aad41e0dabf8005ab04b74c313971b: Status 404 returned error can't find the container with id aaefc27a99b6bf72bffee419f6ba9757c9aad41e0dabf8005ab04b74c313971b Nov 04 05:10:57.713712 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.713697 4581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 05:10:57.972942 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:57.972851 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zx9qc/must-gather-wv8bd" event={"ID":"23567e72-6011-4373-9ce8-3ffe739682f2","Type":"ContainerStarted","Data":"aaefc27a99b6bf72bffee419f6ba9757c9aad41e0dabf8005ab04b74c313971b"} Nov 04 05:10:59.983715 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:59.983680 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zx9qc/must-gather-wv8bd" event={"ID":"23567e72-6011-4373-9ce8-3ffe739682f2","Type":"ContainerStarted","Data":"866cac1dc2611a69b2ef44afa31f2d5743fe77200540091be08a02fd8db3b895"} Nov 04 05:10:59.983715 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:10:59.983721 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zx9qc/must-gather-wv8bd" event={"ID":"23567e72-6011-4373-9ce8-3ffe739682f2","Type":"ContainerStarted","Data":"8f22eea29849fd6b094402635fc72524ff5046aeb2158c9bb430dee4d74b7e72"} Nov 04 05:11:00.098233 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:00.098169 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zx9qc/must-gather-wv8bd" podStartSLOduration=2.81965268 podStartE2EDuration="4.098148724s" totalImagesPullingTime="1.278496044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:10:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:11:00.097461681 +0000 UTC m=+1418.988318702" watchObservedRunningTime="2026-11-04 05:11:00.098148724 +0000 UTC m=+1418.989005746" Nov 04 05:11:04.148556 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:04.148509 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:11:06.597008 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:06.596962 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m5khv_24bc4691-e47f-4f98-ad47-f14654d95d28/global-pull-secret-syncer/0.log" Nov 04 05:11:09.207240 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:09.206981 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" Nov 04 05:11:11.583978 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:11.583681 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-132.us-west-1.compute.internal_e7859ee47f36ca521b3b7f587511c5ea/haproxy/0.log" Nov 04 05:11:15.998419 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:15.998390 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-4bhfh_a19b0538-b992-48a4-8566-960d59838722/kuadrant-console-plugin/0.log" Nov 04 05:11:16.244584 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:16.244553 4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zx9qc/perf-node-gather-daemonset-qbrql"] Nov 04 05:11:16.252222 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:16.252157 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-qbrql" Nov 04 05:11:16.334713 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:16.334677 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b15c7977-e8ba-403a-8e29-0299e67525b5-podres\") pod \"perf-node-gather-daemonset-qbrql\" (UID: \"b15c7977-e8ba-403a-8e29-0299e67525b5\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-qbrql" Nov 04 05:11:16.334912 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:16.334739 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kldpz\" (UniqueName: \"kubernetes.io/projected/b15c7977-e8ba-403a-8e29-0299e67525b5-kube-api-access-kldpz\") pod \"perf-node-gather-daemonset-qbrql\" (UID: \"b15c7977-e8ba-403a-8e29-0299e67525b5\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-qbrql" Nov 04 05:11:16.334912 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:16.334779 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b15c7977-e8ba-403a-8e29-0299e67525b5-lib-modules\") pod \"perf-node-gather-daemonset-qbrql\" (UID: \"b15c7977-e8ba-403a-8e29-0299e67525b5\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-qbrql" Nov 04 05:11:16.334912 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:16.334834 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b15c7977-e8ba-403a-8e29-0299e67525b5-proc\") pod \"perf-node-gather-daemonset-qbrql\" (UID: \"b15c7977-e8ba-403a-8e29-0299e67525b5\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-qbrql" Nov 04 05:11:16.334912 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:16.334858 4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b15c7977-e8ba-403a-8e29-0299e67525b5-sys\") pod \"perf-node-gather-daemonset-qbrql\" (UID: \"b15c7977-e8ba-403a-8e29-0299e67525b5\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-qbrql" Nov 04 05:11:16.340945 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:16.340916 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zx9qc/perf-node-gather-daemonset-qbrql"] Nov 04 05:11:16.436313 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:16.436284 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b15c7977-e8ba-403a-8e29-0299e67525b5-lib-modules\") pod \"perf-node-gather-daemonset-qbrql\" (UID: \"b15c7977-e8ba-403a-8e29-0299e67525b5\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-qbrql" Nov 04 05:11:16.436481 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:16.436341 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b15c7977-e8ba-403a-8e29-0299e67525b5-proc\") pod \"perf-node-gather-daemonset-qbrql\" (UID: \"b15c7977-e8ba-403a-8e29-0299e67525b5\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-qbrql" Nov 04 05:11:16.436481 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:16.436364 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b15c7977-e8ba-403a-8e29-0299e67525b5-sys\") pod \"perf-node-gather-daemonset-qbrql\" (UID: \"b15c7977-e8ba-403a-8e29-0299e67525b5\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-qbrql" Nov 04 05:11:16.436481 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:16.436404 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b15c7977-e8ba-403a-8e29-0299e67525b5-podres\") pod \"perf-node-gather-daemonset-qbrql\" (UID: \"b15c7977-e8ba-403a-8e29-0299e67525b5\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-qbrql" Nov 04 05:11:16.436481 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:16.436438 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b15c7977-e8ba-403a-8e29-0299e67525b5-lib-modules\") pod \"perf-node-gather-daemonset-qbrql\" (UID: \"b15c7977-e8ba-403a-8e29-0299e67525b5\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-qbrql" Nov 04 05:11:16.436481 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:16.436468 4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kldpz\" (UniqueName: \"kubernetes.io/projected/b15c7977-e8ba-403a-8e29-0299e67525b5-kube-api-access-kldpz\") pod \"perf-node-gather-daemonset-qbrql\" (UID: \"b15c7977-e8ba-403a-8e29-0299e67525b5\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-qbrql" Nov 04 05:11:16.436481 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:16.436480 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b15c7977-e8ba-403a-8e29-0299e67525b5-sys\") pod \"perf-node-gather-daemonset-qbrql\" (UID: \"b15c7977-e8ba-403a-8e29-0299e67525b5\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-qbrql" Nov 04 05:11:16.436709 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:16.436516 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b15c7977-e8ba-403a-8e29-0299e67525b5-podres\") pod \"perf-node-gather-daemonset-qbrql\" (UID: \"b15c7977-e8ba-403a-8e29-0299e67525b5\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-qbrql" Nov 04 05:11:16.436709 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:16.436468 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b15c7977-e8ba-403a-8e29-0299e67525b5-proc\") pod \"perf-node-gather-daemonset-qbrql\" (UID: \"b15c7977-e8ba-403a-8e29-0299e67525b5\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-qbrql" Nov 04 05:11:16.493912 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:16.493887 4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kldpz\" (UniqueName: \"kubernetes.io/projected/b15c7977-e8ba-403a-8e29-0299e67525b5-kube-api-access-kldpz\") pod \"perf-node-gather-daemonset-qbrql\" (UID: \"b15c7977-e8ba-403a-8e29-0299e67525b5\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-qbrql" Nov 04 05:11:16.568625 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:16.568550 4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-qbrql" Nov 04 05:11:16.948595 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:16.948570 4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zx9qc/perf-node-gather-daemonset-qbrql"] Nov 04 05:11:16.950530 ip-10-0-141-132 kubenswrapper[4581]: W1104 05:11:16.950502 4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb15c7977_e8ba_403a_8e29_0299e67525b5.slice/crio-5371bb04357f93dfc62ba3e676e14c092279cb9b07a0c038f0d0286bcd5bd405 WatchSource:0}: Error finding container 5371bb04357f93dfc62ba3e676e14c092279cb9b07a0c038f0d0286bcd5bd405: Status 404 returned error can't find the container with id 5371bb04357f93dfc62ba3e676e14c092279cb9b07a0c038f0d0286bcd5bd405 Nov 04 05:11:17.140036 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:17.139969 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-qbrql" event={"ID":"b15c7977-e8ba-403a-8e29-0299e67525b5","Type":"ContainerStarted","Data":"3b9d1f7b61a6727dbbdf96e0e07cf45c9ee7278f7994f2d6c634edc176fad486"} Nov 04 05:11:17.140557 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:17.140049 4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-qbrql" event={"ID":"b15c7977-e8ba-403a-8e29-0299e67525b5","Type":"ContainerStarted","Data":"5371bb04357f93dfc62ba3e676e14c092279cb9b07a0c038f0d0286bcd5bd405"} Nov 04 05:11:17.140557 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:17.140117 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-qbrql" Nov 04 05:11:17.255324 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:17.255235 4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-qbrql" podStartSLOduration=1.255220894 podStartE2EDuration="1.255220894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 05:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:11:17.254141638 +0000 UTC m=+1436.144998659" watchObservedRunningTime="2026-11-04 05:11:17.255220894 +0000 UTC m=+1436.146077913" Nov 04 05:11:17.893222 ip-10-0-141-132 kubenswrapper[4581]: E1104 05:11:17.893188 4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-724sb" containerName="extractor" Nov 04 05:11:20.221274 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:20.221239 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c11b727-b2d3-4c3e-ba45-4335b8af0d2d/alertmanager/0.log" Nov 04 05:11:20.641737 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:20.641708 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c11b727-b2d3-4c3e-ba45-4335b8af0d2d/config-reloader/0.log" Nov 04 05:11:21.063259 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:21.063174 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c11b727-b2d3-4c3e-ba45-4335b8af0d2d/kube-rbac-proxy-web/0.log" Nov 04 05:11:21.486356 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:21.486330 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c11b727-b2d3-4c3e-ba45-4335b8af0d2d/kube-rbac-proxy/0.log" Nov 04 05:11:21.909753 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:21.909725 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c11b727-b2d3-4c3e-ba45-4335b8af0d2d/kube-rbac-proxy-metric/0.log" Nov 04 05:11:22.333719 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:22.333650 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c11b727-b2d3-4c3e-ba45-4335b8af0d2d/prom-label-proxy/0.log" Nov 04 05:11:22.756389 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:22.756353 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c11b727-b2d3-4c3e-ba45-4335b8af0d2d/init-config-reloader/0.log" Nov 04 05:11:23.156379 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:23.156353 4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-qbrql" Nov 04 05:11:23.707556 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:23.707527 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-hklc2_a3d46a1d-9630-4f75-97c8-09302e92d701/kube-state-metrics/0.log" Nov 04 05:11:24.166002 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:24.165971 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-hklc2_a3d46a1d-9630-4f75-97c8-09302e92d701/kube-rbac-proxy-main/0.log" Nov 04 05:11:24.587195 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:24.587113 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-hklc2_a3d46a1d-9630-4f75-97c8-09302e92d701/kube-rbac-proxy-self/0.log" Nov 04 05:11:25.011817 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:25.011792 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7f458b6499-5vx8b_3e1448fd-a504-43b2-b474-63abb5fc7d17/metrics-server/0.log" Nov 04 05:11:28.388452 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:28.388420 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rpwmg_d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c/node-exporter/0.log" Nov 04 05:11:28.815736 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:28.815652 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rpwmg_d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c/kube-rbac-proxy/0.log" Nov 04 05:11:29.240188 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:29.240063 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rpwmg_d016d731-0fb7-4f5a-8f30-3d3e5ee5be6c/init-textfile/0.log" Nov 04 05:11:30.935781 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:30.935739 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_97f1373d-ba83-44f9-bb25-31ada871c58b/prometheus/0.log" Nov 04 05:11:31.357413 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:31.357387 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_97f1373d-ba83-44f9-bb25-31ada871c58b/config-reloader/0.log" Nov 04 05:11:31.778410 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:31.778330 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_97f1373d-ba83-44f9-bb25-31ada871c58b/thanos-sidecar/0.log" Nov 04 05:11:32.201374 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:32.201344 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_97f1373d-ba83-44f9-bb25-31ada871c58b/kube-rbac-proxy-web/0.log" Nov 04 05:11:32.625927 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:32.625901 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_97f1373d-ba83-44f9-bb25-31ada871c58b/kube-rbac-proxy/0.log" Nov 04 05:11:33.048571 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:33.048478 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_97f1373d-ba83-44f9-bb25-31ada871c58b/kube-rbac-proxy-thanos/0.log" Nov 04 05:11:33.471067 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:33.471041 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_97f1373d-ba83-44f9-bb25-31ada871c58b/init-config-reloader/0.log" Nov 04 05:11:33.898138 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:33.898115 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-d9tqz_f1580245-c2af-4604-870c-390f2bdc22c5/prometheus-operator/0.log" Nov 04 05:11:34.316692 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:34.316618 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-d9tqz_f1580245-c2af-4604-870c-390f2bdc22c5/kube-rbac-proxy/0.log" Nov 04 05:11:35.578037 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:35.578004 4581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7csxg_030a8f10-c4b1-45b2-abb8-88ed48e53a80/konnectivity-agent/0.log" limit=1024 Nov 04 05:11:36.425143 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:36.425116 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7976f76979-72mkk_a06cbcc0-f590-4829-8dc0-1d173b2a4c6e/thanos-query/0.log" Nov 04 05:11:36.846693 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:36.846615 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7976f76979-72mkk_a06cbcc0-f590-4829-8dc0-1d173b2a4c6e/kube-rbac-proxy-web/0.log" Nov 04 05:11:37.269616 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:37.269590 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7976f76979-72mkk_a06cbcc0-f590-4829-8dc0-1d173b2a4c6e/kube-rbac-proxy/0.log" Nov 04 05:11:37.693860 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:37.693822 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7976f76979-72mkk_a06cbcc0-f590-4829-8dc0-1d173b2a4c6e/prom-label-proxy/0.log" Nov 04 05:11:38.117250 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:38.117225 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7976f76979-72mkk_a06cbcc0-f590-4829-8dc0-1d173b2a4c6e/kube-rbac-proxy-rules/0.log" Nov 04 05:11:38.538211 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:38.538129 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7976f76979-72mkk_a06cbcc0-f590-4829-8dc0-1d173b2a4c6e/kube-rbac-proxy-metrics/0.log" Nov 04 05:11:39.850554 ip-10-0-141-132 kubenswrapper[4581]: I1104 05:11:39.850515 4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-546d78d5c-g8s92_63a561a1-32d4-41b1-a285-4e82379ac8d4/kube-auth-proxy/0.log"