Sep 05 06:24:22.967140 ip-10-0-141-66 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 05 06:24:22.967151 ip-10-0-141-66 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 05 06:24:22.967159 ip-10-0-141-66 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 05 06:24:22.967411 ip-10-0-141-66 systemd[1]: Failed to start Kubernetes Kubelet. Sep 05 06:24:33.002178 ip-10-0-141-66 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 05 06:24:33.002195 ip-10-0-141-66 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 13697b4ea2a04f5c802c240979b6cc0a -- Sep 05 06:26:15.489626 ip-10-0-141-66 systemd[1]: Starting Kubernetes Kubelet... Sep 05 06:26:15.923185 ip-10-0-141-66 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 05 06:26:15.923185 ip-10-0-141-66 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 05 06:26:15.923185 ip-10-0-141-66 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 05 06:26:15.923185 ip-10-0-141-66 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 05 06:26:15.923185 ip-10-0-141-66 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 05 06:26:15.924356 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.924257 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 05 06:26:15.928970 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.928942 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 05 06:26:15.928970 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.928964 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 05 06:26:15.928970 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.928974 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 05 06:26:15.929187 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.928978 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 05 06:26:15.929187 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.928983 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 05 06:26:15.929187 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.928987 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 05 06:26:15.929187 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.928991 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 05 06:26:15.929187 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.928995 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 05 06:26:15.929187 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929000 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 05 06:26:15.929187 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929004 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 05 06:26:15.929187 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929007 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 05 06:26:15.929187 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929011 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 05 06:26:15.929187 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929015 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 05 06:26:15.929187 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929020 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 05 06:26:15.929187 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929024 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 05 06:26:15.929187 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929027 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 05 06:26:15.929187 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929031 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 05 06:26:15.929187 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929038 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 05 06:26:15.929187 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929043 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 05 06:26:15.929187 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929048 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 05 06:26:15.929187 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929052 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 05 06:26:15.929187 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929057 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 05 06:26:15.929187 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929061 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 05 06:26:15.930020 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929065 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 05 06:26:15.930020 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929069 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 05 06:26:15.930020 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929074 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 05 06:26:15.930020 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929078 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 05 06:26:15.930020 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929082 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 05 06:26:15.930020 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929087 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 05 06:26:15.930020 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929090 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 05 06:26:15.930020 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929095 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 05 06:26:15.930020 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929099 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 05 06:26:15.930020 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929105 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 05 06:26:15.930020 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929117 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 05 06:26:15.930020 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929122 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 05 06:26:15.930020 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929127 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 05 06:26:15.930020 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929133 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 05 06:26:15.930020 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929137 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 05 06:26:15.930020 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929142 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 05 06:26:15.930020 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929146 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 05 06:26:15.930020 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929150 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 05 06:26:15.930020 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929154 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 05 06:26:15.930649 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929159 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 05 06:26:15.930649 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929163 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 05 06:26:15.930649 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929168 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 05 06:26:15.930649 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929172 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 05 06:26:15.930649 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929177 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 05 06:26:15.930649 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929181 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 05 06:26:15.930649 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929185 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 05 06:26:15.930649 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929189 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 05 06:26:15.930649 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929195 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 05 06:26:15.930649 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929199 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 05 06:26:15.930649 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929203 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 05 06:26:15.930649 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929209 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 05 06:26:15.930649 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929214 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 05 06:26:15.930649 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929220 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 05 06:26:15.930649 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929226 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 05 06:26:15.930649 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929233 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 05 06:26:15.930649 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929237 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 05 06:26:15.930649 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929241 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 05 06:26:15.930649 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929246 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 05 06:26:15.931120 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929250 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 05 06:26:15.931120 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929254 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 05 06:26:15.931120 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929258 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 05 06:26:15.931120 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929262 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 05 06:26:15.931120 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929267 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 05 06:26:15.931120 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929271 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 05 06:26:15.931120 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929275 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 05 06:26:15.931120 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929281 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 05 06:26:15.931120 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929285 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 05 06:26:15.931120 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929290 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 05 06:26:15.931120 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929294 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 05 06:26:15.931120 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929298 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 05 06:26:15.931120 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929302 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 05 06:26:15.931120 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929325 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 05 06:26:15.931120 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929329 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 05 06:26:15.931120 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929334 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 05 06:26:15.931120 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929338 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 05 06:26:15.931120 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929342 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 05 06:26:15.931120 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929346 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 05 06:26:15.931120 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929350 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 05 06:26:15.931673 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929355 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 05 06:26:15.931673 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929359 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 05 06:26:15.931673 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929364 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 05 06:26:15.931673 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929369 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 05 06:26:15.931673 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929373 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 05 06:26:15.931673 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929377 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 05 06:26:15.931673 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929382 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 05 06:26:15.931673 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929386 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 05 06:26:15.931673 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929390 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 05 06:26:15.931673 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929394 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 05 06:26:15.931673 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929398 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 05 06:26:15.931673 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929402 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 05 06:26:15.931673 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929406 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 05 06:26:15.931673 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929410 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 05 06:26:15.931673 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929414 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 05 06:26:15.931673 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929419 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 05 06:26:15.931673 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929423 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 05 06:26:15.931673 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.929427 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 05 06:26:15.932605 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932587 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 05 06:26:15.932605 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932606 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 05 06:26:15.932733 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932612 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 05 06:26:15.932733 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932617 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 05 06:26:15.932733 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932622 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 05 06:26:15.932733 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932625 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 05 06:26:15.932733 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932629 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 05 06:26:15.932733 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932633 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 05 06:26:15.932733 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932637 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 05 06:26:15.932733 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932640 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 05 06:26:15.932733 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932644 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 05 06:26:15.932733 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932648 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 05 06:26:15.932733 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932653 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 05 06:26:15.932733 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932657 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 05 06:26:15.932733 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932661 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 05 06:26:15.932733 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932665 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 05 06:26:15.932733 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932668 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 05 06:26:15.932733 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932674 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 05 06:26:15.932733 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932678 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 05 06:26:15.932733 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932682 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 05 06:26:15.932733 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932685 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 05 06:26:15.933589 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932689 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 05 06:26:15.933589 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932693 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 05 06:26:15.933589 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932697 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 05 06:26:15.933589 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932704 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 05 06:26:15.933589 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932709 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 05 06:26:15.933589 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932715 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 05 06:26:15.933589 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932719 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 05 06:26:15.933589 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932724 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 05 06:26:15.933589 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932729 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 05 06:26:15.933589 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932733 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 05 06:26:15.933589 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932738 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 05 06:26:15.933589 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932742 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 05 06:26:15.933589 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932746 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 05 06:26:15.933589 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932750 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 05 06:26:15.933589 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932754 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 05 06:26:15.933589 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932759 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 05 06:26:15.933589 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932762 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 05 06:26:15.933589 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932766 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 05 06:26:15.933589 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932770 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 05 06:26:15.934421 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932774 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 05 06:26:15.934421 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932778 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 05 06:26:15.934421 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932783 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 05 06:26:15.934421 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932786 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 05 06:26:15.934421 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932790 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 05 06:26:15.934421 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932794 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 05 06:26:15.934421 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932798 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 05 06:26:15.934421 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932802 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 05 06:26:15.934421 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932806 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 05 06:26:15.934421 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932810 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 05 06:26:15.934421 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932815 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 05 06:26:15.934421 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932819 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 05 06:26:15.934421 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932823 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 05 06:26:15.934421 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932828 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 05 06:26:15.934421 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932833 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 05 06:26:15.934421 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932837 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 05 06:26:15.934421 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932841 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 05 06:26:15.934421 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932845 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 05 06:26:15.934421 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932848 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 05 06:26:15.934421 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932852 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 05 06:26:15.934421 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932856 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 05 06:26:15.935341 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932860 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 05 06:26:15.935341 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932864 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 05 06:26:15.935341 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932868 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 05 06:26:15.935341 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932873 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 05 06:26:15.935341 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932877 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 05 06:26:15.935341 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932884 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 05 06:26:15.935341 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932890 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 05 06:26:15.935341 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932895 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 05 06:26:15.935341 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932899 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 05 06:26:15.935341 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932904 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 05 06:26:15.935341 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932909 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 05 06:26:15.935341 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932913 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 05 06:26:15.935341 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932918 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 05 06:26:15.935341 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932922 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 05 06:26:15.935341 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932927 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 05 06:26:15.935341 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932931 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 05 06:26:15.935341 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932935 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 05 06:26:15.935341 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932940 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 05 06:26:15.935341 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932944 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 05 06:26:15.936066 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932949 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 05 06:26:15.936066 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932953 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 05 06:26:15.936066 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932959 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 05 06:26:15.936066 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932965 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 05 06:26:15.936066 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932969 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 05 06:26:15.936066 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932974 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 05 06:26:15.936066 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932979 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 05 06:26:15.936066 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932984 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 05 06:26:15.936066 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932988 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 05 06:26:15.936066 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932992 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 05 06:26:15.936066 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.932997 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 05 06:26:15.936066 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933001 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 05 06:26:15.936066 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933005 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 05 06:26:15.936066 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933011 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 05 06:26:15.936066 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933016 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 05 06:26:15.936066 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933021 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 05 06:26:15.936066 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933025 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 05 06:26:15.936066 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933029 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 05 06:26:15.936066 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933033 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 05 06:26:15.936559 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933749 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 05 06:26:15.936559 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933759 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 05 06:26:15.936559 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933765 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 05 06:26:15.936559 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933769 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 05 06:26:15.936559 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933774 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 05 06:26:15.936559 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933779 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 05 06:26:15.936559 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933783 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 05 06:26:15.936559 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933787 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 05 06:26:15.936559 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933792 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 05 06:26:15.936559 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933797 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 05 06:26:15.936559 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933801 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 05 06:26:15.936559 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933805 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 05 06:26:15.936559 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933809 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 05 06:26:15.936559 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933814 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 05 06:26:15.936559 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933818 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 05 06:26:15.936559 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933822 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 05 06:26:15.936559 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933831 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 05 06:26:15.936559 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933836 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 05 06:26:15.936559 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933843 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 05 06:26:15.937099 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933847 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 05 06:26:15.937099 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933851 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 05 06:26:15.937099 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933855 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 05 06:26:15.937099 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933859 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 05 06:26:15.937099 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933863 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 05 06:26:15.937099 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933867 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 05 06:26:15.937099 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933871 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 05 06:26:15.937099 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933876 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 05 06:26:15.937099 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933880 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 05 06:26:15.937099 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933884 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 05 06:26:15.937099 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933889 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 05 06:26:15.937099 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933893 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 05 06:26:15.937099 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933897 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 05 06:26:15.937099 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933901 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 05 06:26:15.937099 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933905 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 05 06:26:15.937099 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933912 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 05 06:26:15.937099 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933918 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 05 06:26:15.937099 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933922 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 05 06:26:15.937099 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933926 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 05 06:26:15.937099 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933930 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 05 06:26:15.937629 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933934 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 05 06:26:15.937629 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933938 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 05 06:26:15.937629 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933943 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 05 06:26:15.937629 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933947 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 05 06:26:15.937629 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933951 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 05 06:26:15.937629 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933955 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 05 06:26:15.937629 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933960 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 05 06:26:15.937629 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933964 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 05 06:26:15.937629 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933968 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 05 06:26:15.937629 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933972 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 05 06:26:15.937629 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933977 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 05 06:26:15.937629 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933983 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 05 06:26:15.937629 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933987 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 05 06:26:15.937629 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933991 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 05 06:26:15.937629 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.933996 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 05 06:26:15.937629 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934000 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 05 06:26:15.937629 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934004 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 05 06:26:15.937629 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934008 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 05 06:26:15.937629 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934012 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 05 06:26:15.937629 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934017 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 05 06:26:15.938242 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934021 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 05 06:26:15.938242 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934025 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 05 06:26:15.938242 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934029 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 05 06:26:15.938242 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934034 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 05 06:26:15.938242 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934038 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 05 06:26:15.938242 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934042 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 05 06:26:15.938242 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934046 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 05 06:26:15.938242 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934051 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 05 06:26:15.938242 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934055 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 05 06:26:15.938242 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934059 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 05 06:26:15.938242 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934063 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 05 06:26:15.938242 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934067 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 05 06:26:15.938242 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934071 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 05 06:26:15.938242 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934075 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 05 06:26:15.938242 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934079 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 05 06:26:15.938242 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934083 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 05 06:26:15.938242 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934088 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 05 06:26:15.938242 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934092 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 05 06:26:15.938242 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934096 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 05 06:26:15.938242 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934100 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 05 06:26:15.938849 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934104 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 05 06:26:15.938849 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934108 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 05 06:26:15.938849 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934112 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 05 06:26:15.938849 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934129 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 05 06:26:15.938849 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934137 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 05 06:26:15.938849 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934144 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 05 06:26:15.938849 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934148 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 05 06:26:15.938849 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934153 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 05 06:26:15.938849 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934157 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 05 06:26:15.938849 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934162 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 05 06:26:15.938849 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934166 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 05 06:26:15.938849 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934170 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 05 06:26:15.938849 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934174 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 05 06:26:15.938849 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934178 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 05 06:26:15.938849 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934182 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 05 06:26:15.938849 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934186 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 05 06:26:15.938849 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934191 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 05 06:26:15.938849 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934197 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 05 06:26:15.938849 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934202 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 05 06:26:15.939328 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.934207 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 05 06:26:15.939328 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934334 2565 flags.go:64] FLAG: --address="0.0.0.0" Sep 05 06:26:15.939328 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934394 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 05 06:26:15.939328 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934408 2565 flags.go:64] FLAG: --anonymous-auth="true" Sep 05 06:26:15.939328 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934416 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 05 06:26:15.939328 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934422 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 05 06:26:15.939328 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934426 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 05 06:26:15.939328 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934433 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 05 06:26:15.939328 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934439 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 05 06:26:15.939328 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934444 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 05 06:26:15.939328 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934449 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 05 06:26:15.939328 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934454 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 05 06:26:15.939328 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934459 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 05 06:26:15.939328 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934465 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 05 06:26:15.939328 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934470 2565 flags.go:64] FLAG: --cgroup-root="" Sep 05 06:26:15.939328 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934474 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 05 06:26:15.939328 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934479 2565 flags.go:64] FLAG: --client-ca-file="" Sep 05 06:26:15.939328 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934492 2565 flags.go:64] FLAG: --cloud-provider="external" Sep 05 06:26:15.939328 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934496 2565 flags.go:64] FLAG: --cluster-dns="[]" Sep 05 06:26:15.939328 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934506 2565 flags.go:64] FLAG: --cluster-domain="" Sep 05 06:26:15.939328 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934511 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 05 06:26:15.939328 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934516 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 05 06:26:15.939328 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934520 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 05 06:26:15.939328 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934526 2565 flags.go:64] FLAG: --container-log-max-files="5" Sep 05 06:26:15.940000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934532 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 05 06:26:15.940000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934536 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 05 06:26:15.940000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934541 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 05 06:26:15.940000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934546 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 05 06:26:15.940000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934551 2565 flags.go:64] FLAG: --contention-profiling="false" Sep 05 06:26:15.940000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934556 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 05 06:26:15.940000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934561 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 05 06:26:15.940000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934566 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 05 06:26:15.940000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934570 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 05 06:26:15.940000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934576 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 05 06:26:15.940000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934582 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 05 06:26:15.940000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934587 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 05 06:26:15.940000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934591 2565 flags.go:64] FLAG: --enable-load-reader="false" Sep 05 06:26:15.940000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934596 2565 flags.go:64] FLAG: --enable-server="true" Sep 05 06:26:15.940000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934601 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 05 06:26:15.940000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934611 2565 flags.go:64] FLAG: --event-burst="100" Sep 05 06:26:15.940000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934616 2565 flags.go:64] FLAG: --event-qps="50" Sep 05 06:26:15.940000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934620 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 05 06:26:15.940000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934625 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 05 06:26:15.940000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934630 2565 flags.go:64] FLAG: --eviction-hard="" Sep 05 06:26:15.940000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934636 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 05 06:26:15.940000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934641 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 05 06:26:15.940000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934645 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 05 06:26:15.940000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934650 2565 flags.go:64] FLAG: --eviction-soft="" Sep 05 06:26:15.940000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934655 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 05 06:26:15.940645 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934659 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 05 06:26:15.940645 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934671 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 05 06:26:15.940645 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934676 2565 flags.go:64] FLAG: --experimental-mounter-path="" Sep 05 06:26:15.940645 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934681 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 05 06:26:15.940645 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934685 2565 flags.go:64] FLAG: --fail-swap-on="true" Sep 05 06:26:15.940645 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934690 2565 flags.go:64] FLAG: --feature-gates="" Sep 05 06:26:15.940645 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934696 2565 flags.go:64] FLAG: --file-check-frequency="20s" Sep 05 06:26:15.940645 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934701 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 05 06:26:15.940645 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934706 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 05 06:26:15.940645 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934711 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 05 06:26:15.940645 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934716 2565 flags.go:64] FLAG: --healthz-port="10248" Sep 05 06:26:15.940645 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934721 2565 flags.go:64] FLAG: --help="false" Sep 05 06:26:15.940645 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934725 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-141-66.ec2.internal" Sep 05 06:26:15.940645 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934730 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 05 06:26:15.940645 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934735 2565 flags.go:64] FLAG: --http-check-frequency="20s" Sep 05 06:26:15.940645 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934740 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 05 06:26:15.940645 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934745 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 05 06:26:15.940645 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934751 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 05 06:26:15.940645 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934756 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 05 06:26:15.940645 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934760 2565 flags.go:64] FLAG: --image-service-endpoint="" Sep 05 06:26:15.940645 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934764 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 05 06:26:15.940645 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934769 2565 flags.go:64] FLAG: --kube-api-burst="100" Sep 05 06:26:15.940645 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934777 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 05 06:26:15.941240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934782 2565 flags.go:64] FLAG: --kube-api-qps="50" Sep 05 06:26:15.941240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934787 2565 flags.go:64] FLAG: --kube-reserved="" Sep 05 06:26:15.941240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934791 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 05 06:26:15.941240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934796 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 05 06:26:15.941240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934801 2565 flags.go:64] FLAG: --kubelet-cgroups="" Sep 05 06:26:15.941240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934806 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 05 06:26:15.941240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934810 2565 flags.go:64] FLAG: --lock-file="" Sep 05 06:26:15.941240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934815 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 05 06:26:15.941240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934819 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 05 06:26:15.941240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934824 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 05 06:26:15.941240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934832 2565 flags.go:64] FLAG: --log-json-split-stream="false" Sep 05 06:26:15.941240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934838 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 05 06:26:15.941240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934842 2565 flags.go:64] FLAG: --log-text-split-stream="false" Sep 05 06:26:15.941240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934847 2565 flags.go:64] FLAG: --logging-format="text" Sep 05 06:26:15.941240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934852 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 05 06:26:15.941240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934857 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 05 06:26:15.941240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934862 2565 flags.go:64] FLAG: --manifest-url="" Sep 05 06:26:15.941240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934866 2565 flags.go:64] FLAG: --manifest-url-header="" Sep 05 06:26:15.941240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934873 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 05 06:26:15.941240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934879 2565 flags.go:64] FLAG: --max-open-files="1000000" Sep 05 06:26:15.941240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934886 2565 flags.go:64] FLAG: --max-pods="110" Sep 05 06:26:15.941240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934891 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 05 06:26:15.941240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934895 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 05 06:26:15.941240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934900 2565 flags.go:64] FLAG: --memory-manager-policy="None" Sep 05 06:26:15.941240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934905 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 05 06:26:15.941860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934910 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 05 06:26:15.941860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934915 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 05 06:26:15.941860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934919 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 05 06:26:15.941860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934931 2565 flags.go:64] FLAG: --node-status-max-images="50" Sep 05 06:26:15.941860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934935 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 05 06:26:15.941860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934940 2565 flags.go:64] FLAG: --oom-score-adj="-999" Sep 05 06:26:15.941860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934947 2565 flags.go:64] FLAG: --pod-cidr="" Sep 05 06:26:15.941860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934952 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 05 06:26:15.941860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934960 2565 flags.go:64] FLAG: --pod-manifest-path="" Sep 05 06:26:15.941860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934965 2565 flags.go:64] FLAG: --pod-max-pids="-1" Sep 05 06:26:15.941860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934970 2565 flags.go:64] FLAG: --pods-per-core="0" Sep 05 06:26:15.941860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934975 2565 flags.go:64] FLAG: --port="10250" Sep 05 06:26:15.941860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934988 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 05 06:26:15.941860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934993 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09443d6f54945ae11" Sep 05 06:26:15.941860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.934998 2565 flags.go:64] FLAG: --qos-reserved="" Sep 05 06:26:15.941860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935003 2565 flags.go:64] FLAG: --read-only-port="10255" Sep 05 06:26:15.941860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935008 2565 flags.go:64] FLAG: --register-node="true" Sep 05 06:26:15.941860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935013 2565 flags.go:64] FLAG: --register-with-taints="" Sep 05 06:26:15.941860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935019 2565 flags.go:64] FLAG: --registry-burst="10" Sep 05 06:26:15.941860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935024 2565 flags.go:64] FLAG: --registry-qps="5" Sep 05 06:26:15.941860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935029 2565 flags.go:64] FLAG: --reserved-cpus="" Sep 05 06:26:15.941860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935033 2565 flags.go:64] FLAG: --reserved-memory="" Sep 05 06:26:15.941860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935042 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 05 06:26:15.941860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935047 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 05 06:26:15.941860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935052 2565 flags.go:64] FLAG: --rotate-certificates="false" Sep 05 06:26:15.942552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935057 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 05 06:26:15.942552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935062 2565 flags.go:64] FLAG: --runonce="false" Sep 05 06:26:15.942552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935067 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 05 06:26:15.942552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935072 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 05 06:26:15.942552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935076 2565 flags.go:64] FLAG: --seccomp-default="false" Sep 05 06:26:15.942552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935083 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 05 06:26:15.942552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935088 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 05 06:26:15.942552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935093 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 05 06:26:15.942552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935098 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 05 06:26:15.942552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935103 2565 flags.go:64] FLAG: --storage-driver-password="root" Sep 05 06:26:15.942552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935107 2565 flags.go:64] FLAG: --storage-driver-secure="false" Sep 05 06:26:15.942552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935112 2565 flags.go:64] FLAG: --storage-driver-table="stats" Sep 05 06:26:15.942552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935117 2565 flags.go:64] FLAG: --storage-driver-user="root" Sep 05 06:26:15.942552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935121 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 05 06:26:15.942552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935128 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 05 06:26:15.942552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935134 2565 flags.go:64] FLAG: --system-cgroups="" Sep 05 06:26:15.942552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935138 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 05 06:26:15.942552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935147 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 05 06:26:15.942552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935152 2565 flags.go:64] FLAG: --tls-cert-file="" Sep 05 06:26:15.942552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935156 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 05 06:26:15.942552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935166 2565 flags.go:64] FLAG: --tls-min-version="" Sep 05 06:26:15.942552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935171 2565 flags.go:64] FLAG: --tls-private-key-file="" Sep 05 06:26:15.942552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935175 2565 flags.go:64] FLAG: --topology-manager-policy="none" Sep 05 06:26:15.942552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935181 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 05 06:26:15.942552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935185 2565 flags.go:64] FLAG: --topology-manager-scope="container" Sep 05 06:26:15.943184 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935190 2565 flags.go:64] FLAG: --v="2" Sep 05 06:26:15.943184 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935196 2565 flags.go:64] FLAG: --version="false" Sep 05 06:26:15.943184 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935203 2565 flags.go:64] FLAG: --vmodule="" Sep 05 06:26:15.943184 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935209 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 05 06:26:15.943184 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.935215 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 05 06:26:15.943184 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935451 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 05 06:26:15.943184 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935462 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 05 06:26:15.943184 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935468 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 05 06:26:15.943184 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935473 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 05 06:26:15.943184 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935478 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 05 06:26:15.943184 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935482 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 05 06:26:15.943184 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935487 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 05 06:26:15.943184 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935494 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 05 06:26:15.943184 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935499 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 05 06:26:15.943184 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935505 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 05 06:26:15.943184 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935512 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 05 06:26:15.943184 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935517 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 05 06:26:15.943184 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935521 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 05 06:26:15.943184 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935526 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 05 06:26:15.943184 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935531 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 05 06:26:15.943184 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935535 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 05 06:26:15.943729 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935542 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 05 06:26:15.943729 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935547 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 05 06:26:15.943729 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935551 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 05 06:26:15.943729 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935556 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 05 06:26:15.943729 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935560 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 05 06:26:15.943729 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935564 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 05 06:26:15.943729 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935569 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 05 06:26:15.943729 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935572 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 05 06:26:15.943729 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935577 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 05 06:26:15.943729 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935581 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 05 06:26:15.943729 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935585 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 05 06:26:15.943729 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935589 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 05 06:26:15.943729 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935593 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 05 06:26:15.943729 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935599 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 05 06:26:15.943729 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935608 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 05 06:26:15.943729 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935613 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 05 06:26:15.943729 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935617 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 05 06:26:15.943729 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935623 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 05 06:26:15.943729 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935629 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 05 06:26:15.943729 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935633 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 05 06:26:15.944226 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935638 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 05 06:26:15.944226 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935642 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 05 06:26:15.944226 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935646 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 05 06:26:15.944226 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935652 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 05 06:26:15.944226 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935656 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 05 06:26:15.944226 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935660 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 05 06:26:15.944226 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935664 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 05 06:26:15.944226 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935669 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 05 06:26:15.944226 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935673 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 05 06:26:15.944226 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935677 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 05 06:26:15.944226 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935681 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 05 06:26:15.944226 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935685 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 05 06:26:15.944226 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935692 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 05 06:26:15.944226 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935697 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 05 06:26:15.944226 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935701 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 05 06:26:15.944226 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935705 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 05 06:26:15.944226 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935710 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 05 06:26:15.944226 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935714 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 05 06:26:15.944226 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935718 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 05 06:26:15.944724 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935722 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 05 06:26:15.944724 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935726 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 05 06:26:15.944724 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935730 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 05 06:26:15.944724 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935735 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 05 06:26:15.944724 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935739 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 05 06:26:15.944724 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935743 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 05 06:26:15.944724 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935747 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 05 06:26:15.944724 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935751 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 05 06:26:15.944724 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935756 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 05 06:26:15.944724 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935760 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 05 06:26:15.944724 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935765 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 05 06:26:15.944724 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935769 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 05 06:26:15.944724 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935773 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 05 06:26:15.944724 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935777 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 05 06:26:15.944724 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935781 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 05 06:26:15.944724 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935785 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 05 06:26:15.944724 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935791 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 05 06:26:15.944724 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935796 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 05 06:26:15.944724 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935800 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 05 06:26:15.944724 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935804 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 05 06:26:15.945202 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935808 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 05 06:26:15.945202 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935813 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 05 06:26:15.945202 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935817 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 05 06:26:15.945202 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935820 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 05 06:26:15.945202 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935824 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 05 06:26:15.945202 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935831 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 05 06:26:15.945202 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935836 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 05 06:26:15.945202 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935840 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 05 06:26:15.945202 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935844 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 05 06:26:15.945202 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935848 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 05 06:26:15.945202 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935852 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 05 06:26:15.945202 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935856 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 05 06:26:15.945202 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935860 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 05 06:26:15.945202 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935864 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 05 06:26:15.945202 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935868 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 05 06:26:15.945202 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935872 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 05 06:26:15.945202 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935876 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 05 06:26:15.945202 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935880 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 05 06:26:15.945202 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935884 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 05 06:26:15.945202 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935888 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 05 06:26:15.945747 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935892 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 05 06:26:15.945747 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935897 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 05 06:26:15.945747 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935901 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 05 06:26:15.945747 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.935905 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 05 06:26:15.945747 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.936613 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 05 06:26:15.945747 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.944348 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 05 06:26:15.945747 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.944365 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 05 06:26:15.945747 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.944387 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 05 06:26:15.945747 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.944393 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 05 06:26:15.945747 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944462 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 05 06:26:15.945747 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944468 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 05 06:26:15.945747 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944471 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 05 06:26:15.945747 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944474 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 05 06:26:15.945747 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944477 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 05 06:26:15.945747 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944480 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 05 06:26:15.946126 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944482 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 05 06:26:15.946126 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944486 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 05 06:26:15.946126 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944488 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 05 06:26:15.946126 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944491 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 05 06:26:15.946126 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944494 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 05 06:26:15.946126 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944496 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 05 06:26:15.946126 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944499 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 05 06:26:15.946126 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944502 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 05 06:26:15.946126 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944505 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 05 06:26:15.946126 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944508 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 05 06:26:15.946126 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944510 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 05 06:26:15.946126 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944513 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 05 06:26:15.946126 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944515 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 05 06:26:15.946126 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944518 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 05 06:26:15.946126 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944521 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 05 06:26:15.946126 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944523 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 05 06:26:15.946126 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944526 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 05 06:26:15.946126 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944528 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 05 06:26:15.946126 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944531 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 05 06:26:15.946126 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944533 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 05 06:26:15.946639 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944536 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 05 06:26:15.946639 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944538 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 05 06:26:15.946639 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944540 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 05 06:26:15.946639 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944545 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 05 06:26:15.946639 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944548 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 05 06:26:15.946639 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944551 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 05 06:26:15.946639 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944553 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 05 06:26:15.946639 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944556 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 05 06:26:15.946639 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944559 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 05 06:26:15.946639 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944561 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 05 06:26:15.946639 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944564 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 05 06:26:15.946639 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944566 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 05 06:26:15.946639 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944569 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 05 06:26:15.946639 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944571 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 05 06:26:15.946639 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944574 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 05 06:26:15.946639 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944576 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 05 06:26:15.946639 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944579 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 05 06:26:15.946639 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944581 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 05 06:26:15.946639 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944583 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 05 06:26:15.946639 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944586 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 05 06:26:15.947156 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944588 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 05 06:26:15.947156 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944591 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 05 06:26:15.947156 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944593 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 05 06:26:15.947156 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944595 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 05 06:26:15.947156 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944598 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 05 06:26:15.947156 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944602 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 05 06:26:15.947156 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944606 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 05 06:26:15.947156 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944609 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 05 06:26:15.947156 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944611 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 05 06:26:15.947156 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944613 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 05 06:26:15.947156 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944616 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 05 06:26:15.947156 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944619 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 05 06:26:15.947156 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944621 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 05 06:26:15.947156 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944624 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 05 06:26:15.947156 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944627 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 05 06:26:15.947156 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944630 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 05 06:26:15.947156 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944633 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 05 06:26:15.947156 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944636 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 05 06:26:15.947156 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944639 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 05 06:26:15.947156 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944642 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 05 06:26:15.947731 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944644 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 05 06:26:15.947731 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944647 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 05 06:26:15.947731 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944650 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 05 06:26:15.947731 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944652 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 05 06:26:15.947731 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944655 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 05 06:26:15.947731 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944657 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 05 06:26:15.947731 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944660 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 05 06:26:15.947731 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944662 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 05 06:26:15.947731 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944665 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 05 06:26:15.947731 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944667 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 05 06:26:15.947731 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944670 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 05 06:26:15.947731 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944672 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 05 06:26:15.947731 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944675 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 05 06:26:15.947731 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944677 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 05 06:26:15.947731 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944680 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 05 06:26:15.947731 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944683 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 05 06:26:15.947731 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944688 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 05 06:26:15.947731 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944691 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 05 06:26:15.947731 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944694 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 05 06:26:15.948195 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944697 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 05 06:26:15.948195 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944700 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 05 06:26:15.948195 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944702 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 05 06:26:15.948195 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944705 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 05 06:26:15.948195 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944708 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 05 06:26:15.948195 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944711 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 05 06:26:15.948195 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944713 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 05 06:26:15.948195 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944716 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 05 06:26:15.948195 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944719 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 05 06:26:15.948195 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944722 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 05 06:26:15.948195 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944724 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 05 06:26:15.948195 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944727 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 05 06:26:15.948195 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944730 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 05 06:26:15.948195 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944732 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 05 06:26:15.948555 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.944738 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 05 06:26:15.948555 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944876 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 05 06:26:15.948555 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944880 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 05 06:26:15.948555 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944883 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 05 06:26:15.948555 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944886 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 05 06:26:15.948555 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944888 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 05 06:26:15.948555 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944891 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 05 06:26:15.948555 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944894 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 05 06:26:15.948555 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944896 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 05 06:26:15.948555 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944899 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 05 06:26:15.948555 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944901 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 05 06:26:15.948555 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944904 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 05 06:26:15.948555 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944906 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 05 06:26:15.948555 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944909 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 05 06:26:15.948555 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944911 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 05 06:26:15.948917 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944914 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 05 06:26:15.948917 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944916 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 05 06:26:15.948917 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944919 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 05 06:26:15.948917 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944922 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 05 06:26:15.948917 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944924 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 05 06:26:15.948917 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944927 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 05 06:26:15.948917 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944929 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 05 06:26:15.948917 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944932 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 05 06:26:15.948917 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944934 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 05 06:26:15.948917 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944937 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 05 06:26:15.948917 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944939 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 05 06:26:15.948917 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944942 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 05 06:26:15.948917 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944945 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 05 06:26:15.948917 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944948 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 05 06:26:15.948917 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944951 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 05 06:26:15.948917 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944953 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 05 06:26:15.948917 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944956 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 05 06:26:15.948917 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944958 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 05 06:26:15.948917 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944961 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 05 06:26:15.948917 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944963 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 05 06:26:15.949404 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944966 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 05 06:26:15.949404 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944968 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 05 06:26:15.949404 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944971 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 05 06:26:15.949404 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944973 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 05 06:26:15.949404 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944976 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 05 06:26:15.949404 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944978 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 05 06:26:15.949404 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944981 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 05 06:26:15.949404 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944983 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 05 06:26:15.949404 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944986 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 05 06:26:15.949404 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944988 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 05 06:26:15.949404 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944991 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 05 06:26:15.949404 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944993 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 05 06:26:15.949404 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944996 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 05 06:26:15.949404 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.944998 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 05 06:26:15.949404 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945001 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 05 06:26:15.949404 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945004 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 05 06:26:15.949404 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945007 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 05 06:26:15.949404 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945009 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 05 06:26:15.949404 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945012 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 05 06:26:15.949404 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945014 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 05 06:26:15.949891 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945017 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 05 06:26:15.949891 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945020 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 05 06:26:15.949891 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945022 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 05 06:26:15.949891 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945025 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 05 06:26:15.949891 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945027 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 05 06:26:15.949891 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945030 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 05 06:26:15.949891 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945032 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 05 06:26:15.949891 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945042 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 05 06:26:15.949891 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945045 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 05 06:26:15.949891 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945048 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 05 06:26:15.949891 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945051 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 05 06:26:15.949891 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945053 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 05 06:26:15.949891 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945056 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 05 06:26:15.949891 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945058 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 05 06:26:15.949891 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945061 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 05 06:26:15.949891 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945063 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 05 06:26:15.949891 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945066 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 05 06:26:15.949891 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945070 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 05 06:26:15.949891 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945073 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 05 06:26:15.950352 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945076 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 05 06:26:15.950352 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945079 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 05 06:26:15.950352 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945082 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 05 06:26:15.950352 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945085 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 05 06:26:15.950352 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945088 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 05 06:26:15.950352 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945090 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 05 06:26:15.950352 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945093 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 05 06:26:15.950352 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945098 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 05 06:26:15.950352 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945101 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 05 06:26:15.950352 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945104 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 05 06:26:15.950352 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945106 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 05 06:26:15.950352 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945109 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 05 06:26:15.950352 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945111 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 05 06:26:15.950352 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945114 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 05 06:26:15.950352 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945116 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 05 06:26:15.950352 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945119 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 05 06:26:15.950352 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945122 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 05 06:26:15.950352 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945124 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 05 06:26:15.950352 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945127 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 05 06:26:15.950352 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945129 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 05 06:26:15.950825 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945132 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 05 06:26:15.950825 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945142 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 05 06:26:15.950825 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945145 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 05 06:26:15.950825 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945148 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 05 06:26:15.950825 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945151 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 05 06:26:15.950825 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:15.945154 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 05 06:26:15.950825 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.945159 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 05 06:26:15.950825 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.945292 2565 server.go:962] "Client rotation is on, will bootstrap in background" Sep 05 06:26:15.950825 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.947490 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 05 06:26:15.950825 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.948493 2565 server.go:1019] "Starting client certificate rotation" Sep 05 06:26:15.950825 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.948589 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 05 06:26:15.950825 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.948644 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 05 06:26:15.951911 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.951898 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 05 06:26:15.955372 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.955350 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 05 06:26:15.965708 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.965689 2565 log.go:25] "Validated CRI v1 runtime API" Sep 05 06:26:15.970521 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.970503 2565 log.go:25] "Validated CRI v1 image API" Sep 05 06:26:15.972811 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.972794 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 05 06:26:15.975949 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.975923 2565 fs.go:135] Filesystem UUIDs: map[261191af-f0c8-40de-9001-1322437b2f52:/dev/nvme0n1p4 34f7b246-2c71-4246-a8b1-938e2f807be6:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Sep 05 06:26:15.976028 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.975946 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 05 06:26:15.982197 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.982074 2565 manager.go:216] Machine: {Timestamp:2026-09-05 06:26:15.9807397 +0000 UTC m=+0.365282815 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3113670 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24c2d5a15181afa78f62c4c4920703 SystemUUID:ec24c2d5-a151-81af-a78f-62c4c4920703 BootID:13697b4e-a2a0-4f5c-802c-240979b6cc0a Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:bb:8a:58:9c:87 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:bb:8a:58:9c:87 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:37:31:ad:46:4f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 05 06:26:15.982197 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.982184 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 05 06:26:15.982384 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.982332 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 05 06:26:15.984783 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.984764 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 05 06:26:15.985722 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.985691 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 05 06:26:15.985895 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.985729 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-66.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 05 06:26:15.986559 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.986546 2565 topology_manager.go:138] "Creating topology manager with none policy" Sep 05 06:26:15.986618 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.986563 2565 container_manager_linux.go:309] "Creating device plugin manager" Sep 05 06:26:15.986618 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.986574 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 05 06:26:15.986618 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.986594 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 05 06:26:15.988014 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.988002 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 05 06:26:15.991710 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.991696 2565 state_mem.go:36] "Initialized new in-memory state store" Sep 05 06:26:15.991881 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.991870 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 05 06:26:15.991938 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.991909 2565 kubelet.go:486] "Attempting to sync node with API server" Sep 05 06:26:15.991938 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.991925 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 05 06:26:15.992025 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.991945 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 05 06:26:15.992025 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.991958 2565 kubelet.go:398] "Adding apiserver pod source" Sep 05 06:26:15.992025 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.991982 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 05 06:26:15.992410 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.992396 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 05 06:26:15.992471 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.992422 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 05 06:26:15.994931 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.994911 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 05 06:26:15.998110 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.998091 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 05 06:26:15.998188 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.998124 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 05 06:26:15.998297 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.998283 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 05 06:26:15.998375 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.998301 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 05 06:26:15.998375 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.998324 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 05 06:26:15.998375 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.998332 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 05 06:26:15.998375 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.998341 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 05 06:26:15.998375 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.998347 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 05 06:26:15.999057 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.999036 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 05 06:26:15.999057 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.999052 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 05 06:26:15.999182 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.999065 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 05 06:26:15.999182 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.999073 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 05 06:26:15.999182 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.999087 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 05 06:26:15.999182 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.999099 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 05 06:26:15.999182 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.999126 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 05 06:26:15.999182 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:15.999140 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 05 06:26:16.003141 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.003120 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-66.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 05 06:26:16.003935 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:16.003902 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 05 06:26:16.004026 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:16.003902 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-66.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 05 06:26:16.004122 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.004108 2565 server.go:1268] "Started kubelet" Sep 05 06:26:16.004242 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.004211 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 05 06:26:16.004363 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.004209 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 05 06:26:16.004363 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.004286 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 05 06:26:16.005075 ip-10-0-141-66 systemd[1]: Started Kubernetes Kubelet. Sep 05 06:26:16.005534 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.005517 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 05 06:26:16.006719 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.006708 2565 server.go:310] "Adding debug handlers to kubelet server" Sep 05 06:26:16.009842 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:16.009820 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 05 06:26:16.010106 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.010088 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 05 06:26:16.010325 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.010078 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 05 06:26:16.010443 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:16.010428 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found" Sep 05 06:26:16.010667 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.010644 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kdvzd" Sep 05 06:26:16.011000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.010960 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 05 06:26:16.011000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.011002 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 05 06:26:16.011144 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.011127 2565 reconstruct.go:97] "Volume reconstruction finished" Sep 05 06:26:16.011144 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.011143 2565 reconciler.go:29] "Reconciler: start to sync state" Sep 05 06:26:16.011251 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.010976 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 05 06:26:16.011491 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.011476 2565 factory.go:153] Registering CRI-O factory Sep 05 06:26:16.011568 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.011493 2565 factory.go:223] Registration of the crio container factory successfully Sep 05 06:26:16.011568 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.011545 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 05 06:26:16.011568 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.011554 2565 factory.go:55] Registering systemd factory Sep 05 06:26:16.011568 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.011563 2565 factory.go:223] Registration of the systemd container factory successfully Sep 05 06:26:16.011753 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.011580 2565 factory.go:103] Registering Raw factory Sep 05 06:26:16.011753 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.011594 2565 manager.go:1196] Started watching for new ooms in manager Sep 05 06:26:16.011753 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:16.010233 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-66.ec2.internal.18d258be36b07f34 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-66.ec2.internal,UID:ip-10-0-141-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-66.ec2.internal,},FirstTimestamp:2026-09-05 06:26:16.004075316 +0000 UTC m=+0.388618431,LastTimestamp:2026-09-05 06:26:16.004075316 +0000 UTC m=+0.388618431,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-66.ec2.internal,}" Sep 05 06:26:16.012003 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.011990 2565 manager.go:349] Starting recovery of all containers Sep 05 06:26:16.016872 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.016687 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kdvzd" Sep 05 06:26:16.020078 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:16.020049 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 05 06:26:16.020186 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:16.020164 2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-66.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 05 06:26:16.020736 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.020714 2565 manager.go:354] Recovery completed Sep 05 06:26:16.022255 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.022191 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 05 06:26:16.026862 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.026848 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 05 06:26:16.028966 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.028950 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientMemory" Sep 05 06:26:16.029043 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.028978 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasNoDiskPressure" Sep 05 06:26:16.029043 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.028988 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientPID" Sep 05 06:26:16.029544 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.029532 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 05 06:26:16.029544 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.029545 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 05 06:26:16.029644 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.029534 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 05 06:26:16.029644 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.029559 2565 state_mem.go:36] "Initialized new in-memory state store" Sep 05 06:26:16.030704 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.030679 2565 policy_none.go:49] "None policy: Start" Sep 05 06:26:16.030704 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.030704 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 05 06:26:16.030839 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.030717 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 05 06:26:16.030839 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.030766 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientMemory" Sep 05 06:26:16.030839 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.030800 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasNoDiskPressure" Sep 05 06:26:16.030839 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.030815 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientPID" Sep 05 06:26:16.031866 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.031854 2565 policy_none.go:47] "Start" Sep 05 06:26:16.067883 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.067858 2565 manager.go:337] "Starting Device Plugin manager" Sep 05 06:26:16.067986 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:16.067943 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 05 06:26:16.067986 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.067959 2565 server.go:87] "Starting device plugin registration server" Sep 05 06:26:16.068323 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.068292 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 05 06:26:16.068390 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.068324 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 05 06:26:16.068518 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.068502 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 05 06:26:16.068633 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.068613 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 05 06:26:16.068633 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.068623 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 05 06:26:16.069140 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:16.069118 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 05 06:26:16.069248 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:16.069172 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-66.ec2.internal\" not found" Sep 05 06:26:16.158987 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.158946 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 05 06:26:16.160917 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.159015 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 05 06:26:16.160917 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.159044 2565 kubelet.go:2448] "Starting kubelet main sync loop" Sep 05 06:26:16.160917 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:16.159081 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 05 06:26:16.161302 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.161287 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 05 06:26:16.169364 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.169345 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 05 06:26:16.170192 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.170173 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientMemory" Sep 05 06:26:16.170270 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.170210 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasNoDiskPressure" Sep 05 06:26:16.170270 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.170220 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientPID" Sep 05 06:26:16.170270 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.170242 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-66.ec2.internal" Sep 05 06:26:16.181263 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.181190 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-66.ec2.internal" Sep 05 06:26:16.181263 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:16.181213 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-66.ec2.internal\": node \"ip-10-0-141-66.ec2.internal\" not found" Sep 05 06:26:16.199654 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:16.199631 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found" Sep 05 06:26:16.259419 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.259375 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal"] Sep 05 06:26:16.259565 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.259463 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 05 06:26:16.261251 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.261235 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientMemory" Sep 05 06:26:16.261333 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.261269 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasNoDiskPressure" Sep 05 06:26:16.261333 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.261280 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientPID" Sep 05 06:26:16.263384 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.263371 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 05 06:26:16.263512 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.263499 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" Sep 05 06:26:16.263545 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.263538 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 05 06:26:16.264086 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.264070 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientMemory" Sep 05 06:26:16.264161 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.264098 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasNoDiskPressure" Sep 05 06:26:16.264161 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.264111 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientPID" Sep 05 06:26:16.264237 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.264099 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientMemory" Sep 05 06:26:16.264275 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.264249 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasNoDiskPressure" Sep 05 06:26:16.264275 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.264259 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientPID" Sep 05 06:26:16.266229 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.266213 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal" Sep 05 06:26:16.266285 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.266242 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 05 06:26:16.266955 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.266935 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientMemory" Sep 05 06:26:16.267058 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.266973 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasNoDiskPressure" Sep 05 06:26:16.267058 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.266997 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientPID" Sep 05 06:26:16.280919 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:16.280900 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-66.ec2.internal\" not found" node="ip-10-0-141-66.ec2.internal" Sep 05 06:26:16.284464 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:16.284444 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-66.ec2.internal\" not found" node="ip-10-0-141-66.ec2.internal" Sep 05 06:26:16.299902 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:16.299884 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found" Sep 05 06:26:16.313302 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.313280 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d39c298c5184aab44649905d0fcf62a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal\" (UID: \"0d39c298c5184aab44649905d0fcf62a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" Sep 05 06:26:16.313422 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.313322 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6a158339afd761bc08604ebeb5aa4b2a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-66.ec2.internal\" (UID: \"6a158339afd761bc08604ebeb5aa4b2a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal" Sep 05 06:26:16.313422 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.313342 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0d39c298c5184aab44649905d0fcf62a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal\" (UID: \"0d39c298c5184aab44649905d0fcf62a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" Sep 05 06:26:16.400659 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:16.400633 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found" Sep 05 06:26:16.413732 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.413709 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0d39c298c5184aab44649905d0fcf62a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal\" (UID: \"0d39c298c5184aab44649905d0fcf62a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" Sep 05 06:26:16.413791 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.413738 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d39c298c5184aab44649905d0fcf62a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal\" (UID: \"0d39c298c5184aab44649905d0fcf62a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" Sep 05 06:26:16.413791 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.413756 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6a158339afd761bc08604ebeb5aa4b2a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-66.ec2.internal\" (UID: \"6a158339afd761bc08604ebeb5aa4b2a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal" Sep 05 06:26:16.413791 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.413785 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6a158339afd761bc08604ebeb5aa4b2a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-66.ec2.internal\" (UID: \"6a158339afd761bc08604ebeb5aa4b2a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal" Sep 05 06:26:16.413882 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.413816 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0d39c298c5184aab44649905d0fcf62a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal\" (UID: \"0d39c298c5184aab44649905d0fcf62a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" Sep 05 06:26:16.413882 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.413816 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d39c298c5184aab44649905d0fcf62a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal\" (UID: \"0d39c298c5184aab44649905d0fcf62a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" Sep 05 06:26:16.501205 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:16.501128 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found" Sep 05 06:26:16.583638 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.583609 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" Sep 05 06:26:16.587159 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.587145 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal" Sep 05 06:26:16.601804 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:16.601778 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found" Sep 05 06:26:16.702830 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:16.702780 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found" Sep 05 06:26:16.803427 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:16.803341 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found" Sep 05 06:26:16.829056 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.829034 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 05 06:26:16.886289 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.886257 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 05 06:26:16.904475 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:16.904437 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found" Sep 05 06:26:16.947842 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.947799 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 05 06:26:16.948413 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.947998 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 05 06:26:16.948413 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.948000 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 05 06:26:16.948413 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:16.948003 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 05 06:26:17.004714 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:17.004676 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found" Sep 05 06:26:17.019286 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:17.019250 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-04 06:21:16 +0000 UTC" deadline="2028-02-01 08:09:22.129671514 +0000 UTC" Sep 05 06:26:17.019397 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:17.019285 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12337h43m5.110394134s" Sep 05 06:26:17.105425 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:17.105359 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found" Sep 05 06:26:17.206282 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:17.206245 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found" Sep 05 06:26:17.249913 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:17.249869 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a158339afd761bc08604ebeb5aa4b2a.slice/crio-8f4b31a15c33aaf713d35be05b4ab6bae938951cf987a28fc41e8d2785c6e2e9 WatchSource:0}: Error finding container 8f4b31a15c33aaf713d35be05b4ab6bae938951cf987a28fc41e8d2785c6e2e9: Status 404 returned error can't find the container with id 8f4b31a15c33aaf713d35be05b4ab6bae938951cf987a28fc41e8d2785c6e2e9 Sep 05 06:26:17.250446 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:17.250416 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d39c298c5184aab44649905d0fcf62a.slice/crio-696fa540d377364e659563117ebff9933acb3f3498f948817def309b697fb9b6 WatchSource:0}: Error finding container 696fa540d377364e659563117ebff9933acb3f3498f948817def309b697fb9b6: Status 404 returned error can't find the container with id 696fa540d377364e659563117ebff9933acb3f3498f948817def309b697fb9b6 Sep 05 06:26:17.256065 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:17.256049 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 05 06:26:17.307924 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:17.307870 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found" Sep 05 06:26:17.414456 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:17.414391 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found" Sep 05 06:26:17.515405 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:17.515372 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found" Sep 05 06:26:17.564473 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:17.564449 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 05 06:26:17.612648 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:17.612602 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" Sep 05 06:26:17.629008 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:17.628980 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 05 06:26:17.630120 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:17.630106 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal" Sep 05 06:26:17.640874 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:17.640849 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 05 06:26:17.993454 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:17.993192 2565 apiserver.go:52] "Watching apiserver" Sep 05 06:26:18.002971 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.002939 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 05 06:26:18.007001 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.006427 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal","openshift-multus/multus-additional-cni-plugins-ml7g5","openshift-cluster-node-tuning-operator/tuned-dtsh9","openshift-multus/multus-xg8xw","openshift-multus/network-metrics-daemon-545p9","openshift-network-diagnostics/network-check-target-nccpk","openshift-network-operator/iptables-alerter-gf7px","openshift-ovn-kubernetes/ovnkube-node-6mvpl","kube-system/global-pull-secret-syncer-qzpdk","kube-system/konnectivity-agent-zqj4q","kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864","openshift-dns/node-resolver-dzlmk","openshift-image-registry/node-ca-m7rvs"] Sep 05 06:26:18.009274 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.009247 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gf7px" Sep 05 06:26:18.010911 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.010885 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 05 06:26:18.012046 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.011681 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dzlmk" Sep 05 06:26:18.016412 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.013416 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 05 06:26:18.016412 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.013639 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kz7bm\"" Sep 05 06:26:18.016412 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.013813 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 05 06:26:18.016412 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.013975 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 05 06:26:18.016412 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.014152 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.016932 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.016912 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lv4l9\"" Sep 05 06:26:18.017986 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.017157 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 05 06:26:18.017986 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.017525 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 05 06:26:18.017986 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.017851 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.019627 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.018413 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 05 06:26:18.019627 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.018627 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 05 06:26:18.019627 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.018836 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-l9cqq\"" Sep 05 06:26:18.021418 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.020308 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:18.021418 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:18.020423 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-545p9" podUID="9103306e-ef91-4539-b914-8c3cdb23ea94" Sep 05 06:26:18.021418 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.021258 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:18.021418 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:18.021345 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nccpk" podUID="88f016f8-99a0-4560-b4e2-2abd92c5cf90" Sep 05 06:26:18.022994 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.022966 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-etc-sysctl-conf\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.023107 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.023001 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-lib-modules\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.023107 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.023028 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a1707699-6ecc-4982-9a8e-b1e22f0e751a-iptables-alerter-script\") pod \"iptables-alerter-gf7px\" (UID: \"a1707699-6ecc-4982-9a8e-b1e22f0e751a\") " pod="openshift-network-operator/iptables-alerter-gf7px" Sep 05 06:26:18.023107 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.023054 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-run\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.023107 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.023077 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-etc-tuned\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.023320 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.023105 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf5fn\" (UniqueName: \"kubernetes.io/projected/59539926-4278-4795-8ff8-f3482b69df83-kube-api-access-wf5fn\") pod \"node-resolver-dzlmk\" (UID: \"59539926-4278-4795-8ff8-f3482b69df83\") " pod="openshift-dns/node-resolver-dzlmk" Sep 05 06:26:18.023320 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.023129 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-etc-modprobe-d\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.023320 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.023155 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-etc-systemd\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.023320 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.023180 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-sys\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.023320 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.023209 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-var-lib-kubelet\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.023320 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.023230 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-tmp\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.023320 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.023254 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj2db\" (UniqueName: \"kubernetes.io/projected/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-kube-api-access-rj2db\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.023320 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.023282 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1707699-6ecc-4982-9a8e-b1e22f0e751a-host-slash\") pod \"iptables-alerter-gf7px\" (UID: \"a1707699-6ecc-4982-9a8e-b1e22f0e751a\") " pod="openshift-network-operator/iptables-alerter-gf7px" Sep 05 06:26:18.023320 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.023319 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbhzf\" (UniqueName: \"kubernetes.io/projected/a1707699-6ecc-4982-9a8e-b1e22f0e751a-kube-api-access-xbhzf\") pod \"iptables-alerter-gf7px\" (UID: \"a1707699-6ecc-4982-9a8e-b1e22f0e751a\") " pod="openshift-network-operator/iptables-alerter-gf7px" Sep 05 06:26:18.023782 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.023342 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59539926-4278-4795-8ff8-f3482b69df83-hosts-file\") pod \"node-resolver-dzlmk\" (UID: \"59539926-4278-4795-8ff8-f3482b69df83\") " pod="openshift-dns/node-resolver-dzlmk" Sep 05 06:26:18.023782 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.023385 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-etc-kubernetes\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.023782 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.023428 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-host\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.023782 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.023496 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59539926-4278-4795-8ff8-f3482b69df83-tmp-dir\") pod \"node-resolver-dzlmk\" (UID: \"59539926-4278-4795-8ff8-f3482b69df83\") " pod="openshift-dns/node-resolver-dzlmk" Sep 05 06:26:18.023782 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.023523 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-etc-sysconfig\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.023782 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.023562 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-etc-sysctl-d\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.023782 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.022975 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 05 06:26:18.025009 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.024190 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:18.025009 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:18.024258 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzpdk" podUID="8a3094f3-d739-47e4-b8a0-49335c888b12" Sep 05 06:26:18.025009 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.024406 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 05 06:26:18.025009 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.024543 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 05 06:26:18.025009 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.024710 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7pqgb\"" Sep 05 06:26:18.025009 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.024755 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 05 06:26:18.027583 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.027567 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m7rvs" Sep 05 06:26:18.030517 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.030496 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ml7g5" Sep 05 06:26:18.032492 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.032471 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 05 06:26:18.032778 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.032760 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 05 06:26:18.032854 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.032844 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zqj4q" Sep 05 06:26:18.033258 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.033021 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-h7dgb\"" Sep 05 06:26:18.034067 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.033477 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 05 06:26:18.034067 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.033745 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 05 06:26:18.035916 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.035413 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 05 06:26:18.035916 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.035665 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 05 06:26:18.035916 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.035894 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.038293 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.038040 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" Sep 05 06:26:18.038293 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.038135 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gsqzl\"" Sep 05 06:26:18.039089 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.039071 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 05 06:26:18.039221 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.039205 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 05 06:26:18.039310 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.039295 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2gjp5\"" Sep 05 06:26:18.040522 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.040503 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 05 06:26:18.041244 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.041224 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 05 06:26:18.041613 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.041430 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8x8kl\"" Sep 05 06:26:18.041613 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.041469 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 05 06:26:18.042220 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.042196 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 05 06:26:18.042358 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.042204 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 05 06:26:18.042589 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.042555 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 05 06:26:18.044838 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.044813 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 05 06:26:18.045032 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.045015 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 05 06:26:18.045397 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.045368 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-njcnk\"" Sep 05 06:26:18.045542 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.045526 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 05 06:26:18.051240 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.051206 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8qd77" Sep 05 06:26:18.060767 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.060698 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8qd77" Sep 05 06:26:18.112546 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.112513 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 05 06:26:18.125979 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.124444 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5de80d59-44e0-48d2-88de-e51fdc1bd19e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ml7g5\" (UID: \"5de80d59-44e0-48d2-88de-e51fdc1bd19e\") " pod="openshift-multus/multus-additional-cni-plugins-ml7g5" Sep 05 06:26:18.125979 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.124498 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-etc-sysctl-d\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.125979 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.124575 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-etc-sysctl-conf\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.125979 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.124594 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-lib-modules\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.125979 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.124613 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-system-cni-dir\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.125979 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.124643 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-cnibin\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.125979 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.124663 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-etc-kubernetes\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.125979 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.124678 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5de80d59-44e0-48d2-88de-e51fdc1bd19e-system-cni-dir\") pod \"multus-additional-cni-plugins-ml7g5\" (UID: \"5de80d59-44e0-48d2-88de-e51fdc1bd19e\") " pod="openshift-multus/multus-additional-cni-plugins-ml7g5" Sep 05 06:26:18.125979 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.124694 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60296196-88ed-4a44-a893-d7715babb5f8-multus-daemon-config\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.125979 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.124705 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-etc-sysctl-d\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.125979 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.124724 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5de80d59-44e0-48d2-88de-e51fdc1bd19e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ml7g5\" (UID: \"5de80d59-44e0-48d2-88de-e51fdc1bd19e\") " pod="openshift-multus/multus-additional-cni-plugins-ml7g5" Sep 05 06:26:18.125979 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.124765 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-host-slash\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.125979 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.124793 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-log-socket\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.125979 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.124819 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-host-cni-bin\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.125979 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.124837 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-host-cni-netd\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.125979 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.124864 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf5fn\" (UniqueName: \"kubernetes.io/projected/59539926-4278-4795-8ff8-f3482b69df83-kube-api-access-wf5fn\") pod \"node-resolver-dzlmk\" (UID: \"59539926-4278-4795-8ff8-f3482b69df83\") " pod="openshift-dns/node-resolver-dzlmk" Sep 05 06:26:18.125979 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.124869 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-etc-sysctl-conf\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.127129 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.124881 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-etc-systemd\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.127129 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.124898 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-var-lib-kubelet\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.127129 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.124913 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-tmp\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.127129 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.124937 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj2db\" (UniqueName: \"kubernetes.io/projected/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-kube-api-access-rj2db\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.127129 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.124960 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-host-var-lib-kubelet\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.127129 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.124979 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5de80d59-44e0-48d2-88de-e51fdc1bd19e-cni-binary-copy\") pod \"multus-additional-cni-plugins-ml7g5\" (UID: \"5de80d59-44e0-48d2-88de-e51fdc1bd19e\") " pod="openshift-multus/multus-additional-cni-plugins-ml7g5" Sep 05 06:26:18.127129 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.124999 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs\") pod \"network-metrics-daemon-545p9\" (UID: \"9103306e-ef91-4539-b914-8c3cdb23ea94\") " pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:18.127129 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125014 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-lib-modules\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.127129 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125015 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbhzf\" (UniqueName: \"kubernetes.io/projected/a1707699-6ecc-4982-9a8e-b1e22f0e751a-kube-api-access-xbhzf\") pod \"iptables-alerter-gf7px\" (UID: \"a1707699-6ecc-4982-9a8e-b1e22f0e751a\") " pod="openshift-network-operator/iptables-alerter-gf7px" Sep 05 06:26:18.127129 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125042 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-host\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.127129 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125059 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-host-run-multus-certs\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.127129 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125079 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-etc-openvswitch\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.127129 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125111 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-run-ovn\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.127129 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125145 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/000daeb0-4f4d-497e-a81e-f7457115f0e9-ovnkube-script-lib\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.127129 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125171 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb143a2d-2aaa-41ee-b476-2a03fa70ea4f-sys-fs\") pod \"aws-ebs-csi-driver-node-qw864\" (UID: \"fb143a2d-2aaa-41ee-b476-2a03fa70ea4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" Sep 05 06:26:18.127129 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125208 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-host-var-lib-cni-bin\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.127129 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125223 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-hostroot\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.128022 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125249 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb143a2d-2aaa-41ee-b476-2a03fa70ea4f-etc-selinux\") pod \"aws-ebs-csi-driver-node-qw864\" (UID: \"fb143a2d-2aaa-41ee-b476-2a03fa70ea4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" Sep 05 06:26:18.128022 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125274 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-host-kubelet\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.128022 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125320 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-node-log\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.128022 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125465 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-host\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.128022 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125499 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l87jb\" (UniqueName: \"kubernetes.io/projected/000daeb0-4f4d-497e-a81e-f7457115f0e9-kube-api-access-l87jb\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.128022 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125530 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a1707699-6ecc-4982-9a8e-b1e22f0e751a-iptables-alerter-script\") pod \"iptables-alerter-gf7px\" (UID: \"a1707699-6ecc-4982-9a8e-b1e22f0e751a\") " pod="openshift-network-operator/iptables-alerter-gf7px" Sep 05 06:26:18.128022 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125555 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-run\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.128022 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125582 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-etc-tuned\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.128022 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125609 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60296196-88ed-4a44-a893-d7715babb5f8-cni-binary-copy\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.128022 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125638 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-host-var-lib-cni-multus\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.128022 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125671 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5de80d59-44e0-48d2-88de-e51fdc1bd19e-cnibin\") pod \"multus-additional-cni-plugins-ml7g5\" (UID: \"5de80d59-44e0-48d2-88de-e51fdc1bd19e\") " pod="openshift-multus/multus-additional-cni-plugins-ml7g5" Sep 05 06:26:18.128022 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125696 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-run-systemd\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.128022 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125712 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-etc-systemd\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.128022 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125726 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/000daeb0-4f4d-497e-a81e-f7457115f0e9-ovn-node-metrics-cert\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.128022 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125756 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-host-run-k8s-cni-cncf-io\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.128022 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125759 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-var-lib-kubelet\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.128022 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125781 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60b02f38-b310-4273-bc86-e3eb032c1cd4-host\") pod \"node-ca-m7rvs\" (UID: \"60b02f38-b310-4273-bc86-e3eb032c1cd4\") " pod="openshift-image-registry/node-ca-m7rvs" Sep 05 06:26:18.128901 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.125821 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-run\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.128901 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.126342 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb143a2d-2aaa-41ee-b476-2a03fa70ea4f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qw864\" (UID: \"fb143a2d-2aaa-41ee-b476-2a03fa70ea4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" Sep 05 06:26:18.128901 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.126419 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a3094f3-d739-47e4-b8a0-49335c888b12-dbus\") pod \"global-pull-secret-syncer-qzpdk\" (UID: \"8a3094f3-d739-47e4-b8a0-49335c888b12\") " pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:18.128901 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.126459 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-var-lib-openvswitch\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.128901 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.126487 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt4sx\" (UniqueName: \"kubernetes.io/projected/60296196-88ed-4a44-a893-d7715babb5f8-kube-api-access-vt4sx\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.128901 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.126518 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a1707699-6ecc-4982-9a8e-b1e22f0e751a-iptables-alerter-script\") pod \"iptables-alerter-gf7px\" (UID: \"a1707699-6ecc-4982-9a8e-b1e22f0e751a\") " pod="openshift-network-operator/iptables-alerter-gf7px" Sep 05 06:26:18.128901 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.126542 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb143a2d-2aaa-41ee-b476-2a03fa70ea4f-socket-dir\") pod \"aws-ebs-csi-driver-node-qw864\" (UID: \"fb143a2d-2aaa-41ee-b476-2a03fa70ea4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" Sep 05 06:26:18.128901 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.126582 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-multus-cni-dir\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.128901 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.126621 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ch4c\" (UniqueName: \"kubernetes.io/projected/88f016f8-99a0-4560-b4e2-2abd92c5cf90-kube-api-access-8ch4c\") pod \"network-check-target-nccpk\" (UID: \"88f016f8-99a0-4560-b4e2-2abd92c5cf90\") " pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:18.128901 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.126667 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/000daeb0-4f4d-497e-a81e-f7457115f0e9-ovnkube-config\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.128901 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.126714 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-host-run-netns\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.128901 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.126781 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb143a2d-2aaa-41ee-b476-2a03fa70ea4f-registration-dir\") pod \"aws-ebs-csi-driver-node-qw864\" (UID: \"fb143a2d-2aaa-41ee-b476-2a03fa70ea4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" Sep 05 06:26:18.128901 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.126809 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb143a2d-2aaa-41ee-b476-2a03fa70ea4f-device-dir\") pod \"aws-ebs-csi-driver-node-qw864\" (UID: \"fb143a2d-2aaa-41ee-b476-2a03fa70ea4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" Sep 05 06:26:18.128901 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.126861 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhftv\" (UniqueName: \"kubernetes.io/projected/fb143a2d-2aaa-41ee-b476-2a03fa70ea4f-kube-api-access-fhftv\") pod \"aws-ebs-csi-driver-node-qw864\" (UID: \"fb143a2d-2aaa-41ee-b476-2a03fa70ea4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" Sep 05 06:26:18.128901 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.126896 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-systemd-units\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.128901 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.126927 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.130980 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.126960 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1707699-6ecc-4982-9a8e-b1e22f0e751a-host-slash\") pod \"iptables-alerter-gf7px\" (UID: \"a1707699-6ecc-4982-9a8e-b1e22f0e751a\") " pod="openshift-network-operator/iptables-alerter-gf7px" Sep 05 06:26:18.130980 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.126989 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59539926-4278-4795-8ff8-f3482b69df83-hosts-file\") pod \"node-resolver-dzlmk\" (UID: \"59539926-4278-4795-8ff8-f3482b69df83\") " pod="openshift-dns/node-resolver-dzlmk" Sep 05 06:26:18.130980 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127022 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1707699-6ecc-4982-9a8e-b1e22f0e751a-host-slash\") pod \"iptables-alerter-gf7px\" (UID: \"a1707699-6ecc-4982-9a8e-b1e22f0e751a\") " pod="openshift-network-operator/iptables-alerter-gf7px" Sep 05 06:26:18.130980 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127055 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aee7de64-f537-45d5-b9f9-202d8ebc279c-agent-certs\") pod \"konnectivity-agent-zqj4q\" (UID: \"aee7de64-f537-45d5-b9f9-202d8ebc279c\") " pod="kube-system/konnectivity-agent-zqj4q" Sep 05 06:26:18.130980 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127078 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59539926-4278-4795-8ff8-f3482b69df83-hosts-file\") pod \"node-resolver-dzlmk\" (UID: \"59539926-4278-4795-8ff8-f3482b69df83\") " pod="openshift-dns/node-resolver-dzlmk" Sep 05 06:26:18.130980 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127106 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-host-run-netns\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.130980 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127141 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59539926-4278-4795-8ff8-f3482b69df83-tmp-dir\") pod \"node-resolver-dzlmk\" (UID: \"59539926-4278-4795-8ff8-f3482b69df83\") " pod="openshift-dns/node-resolver-dzlmk" Sep 05 06:26:18.130980 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127177 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-etc-sysconfig\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.130980 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127206 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-multus-conf-dir\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.130980 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127238 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-etc-sysconfig\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.130980 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127275 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3094f3-d739-47e4-b8a0-49335c888b12-original-pull-secret\") pod \"global-pull-secret-syncer-qzpdk\" (UID: \"8a3094f3-d739-47e4-b8a0-49335c888b12\") " pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:18.130980 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127305 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2gz5\" (UniqueName: \"kubernetes.io/projected/60b02f38-b310-4273-bc86-e3eb032c1cd4-kube-api-access-g2gz5\") pod \"node-ca-m7rvs\" (UID: \"60b02f38-b310-4273-bc86-e3eb032c1cd4\") " pod="openshift-image-registry/node-ca-m7rvs" Sep 05 06:26:18.130980 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127361 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ksks\" (UniqueName: \"kubernetes.io/projected/5de80d59-44e0-48d2-88de-e51fdc1bd19e-kube-api-access-7ksks\") pod \"multus-additional-cni-plugins-ml7g5\" (UID: \"5de80d59-44e0-48d2-88de-e51fdc1bd19e\") " pod="openshift-multus/multus-additional-cni-plugins-ml7g5" Sep 05 06:26:18.130980 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127391 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aee7de64-f537-45d5-b9f9-202d8ebc279c-konnectivity-ca\") pod \"konnectivity-agent-zqj4q\" (UID: \"aee7de64-f537-45d5-b9f9-202d8ebc279c\") " pod="kube-system/konnectivity-agent-zqj4q" Sep 05 06:26:18.130980 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127443 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.130980 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127472 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59539926-4278-4795-8ff8-f3482b69df83-tmp-dir\") pod \"node-resolver-dzlmk\" (UID: \"59539926-4278-4795-8ff8-f3482b69df83\") " pod="openshift-dns/node-resolver-dzlmk" Sep 05 06:26:18.130980 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127479 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvskd\" (UniqueName: \"kubernetes.io/projected/9103306e-ef91-4539-b914-8c3cdb23ea94-kube-api-access-bvskd\") pod \"network-metrics-daemon-545p9\" (UID: \"9103306e-ef91-4539-b914-8c3cdb23ea94\") " pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:18.131963 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127514 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/000daeb0-4f4d-497e-a81e-f7457115f0e9-env-overrides\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.131963 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127546 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-etc-modprobe-d\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.131963 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127609 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-sys\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.131963 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127642 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5de80d59-44e0-48d2-88de-e51fdc1bd19e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ml7g5\" (UID: \"5de80d59-44e0-48d2-88de-e51fdc1bd19e\") " pod="openshift-multus/multus-additional-cni-plugins-ml7g5" Sep 05 06:26:18.131963 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127653 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-etc-modprobe-d\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.131963 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127666 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-run-openvswitch\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.131963 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127699 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-etc-kubernetes\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.131963 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127720 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-sys\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.131963 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127724 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-multus-socket-dir-parent\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.131963 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127761 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a3094f3-d739-47e4-b8a0-49335c888b12-kubelet-config\") pod \"global-pull-secret-syncer-qzpdk\" (UID: \"8a3094f3-d739-47e4-b8a0-49335c888b12\") " pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:18.131963 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127827 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/60b02f38-b310-4273-bc86-e3eb032c1cd4-serviceca\") pod \"node-ca-m7rvs\" (UID: \"60b02f38-b310-4273-bc86-e3eb032c1cd4\") " pod="openshift-image-registry/node-ca-m7rvs" Sep 05 06:26:18.131963 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.127837 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-etc-kubernetes\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.131963 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.130035 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 05 06:26:18.136766 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.136623 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-etc-tuned\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.136884 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.136795 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-tmp\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.139624 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.139505 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj2db\" (UniqueName: \"kubernetes.io/projected/e8d3234f-c67d-4d69-9c79-e0f46c361ae0-kube-api-access-rj2db\") pod \"tuned-dtsh9\" (UID: \"e8d3234f-c67d-4d69-9c79-e0f46c361ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.140649 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.140626 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbhzf\" (UniqueName: \"kubernetes.io/projected/a1707699-6ecc-4982-9a8e-b1e22f0e751a-kube-api-access-xbhzf\") pod \"iptables-alerter-gf7px\" (UID: \"a1707699-6ecc-4982-9a8e-b1e22f0e751a\") " pod="openshift-network-operator/iptables-alerter-gf7px" Sep 05 06:26:18.142241 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.142216 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf5fn\" (UniqueName: \"kubernetes.io/projected/59539926-4278-4795-8ff8-f3482b69df83-kube-api-access-wf5fn\") pod \"node-resolver-dzlmk\" (UID: \"59539926-4278-4795-8ff8-f3482b69df83\") " pod="openshift-dns/node-resolver-dzlmk" Sep 05 06:26:18.165535 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.165403 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" event={"ID":"0d39c298c5184aab44649905d0fcf62a","Type":"ContainerStarted","Data":"696fa540d377364e659563117ebff9933acb3f3498f948817def309b697fb9b6"} Sep 05 06:26:18.167131 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.167101 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal" event={"ID":"6a158339afd761bc08604ebeb5aa4b2a","Type":"ContainerStarted","Data":"8f4b31a15c33aaf713d35be05b4ab6bae938951cf987a28fc41e8d2785c6e2e9"} Sep 05 06:26:18.228680 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.228643 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60296196-88ed-4a44-a893-d7715babb5f8-cni-binary-copy\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.228680 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.228685 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-host-var-lib-cni-multus\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.228907 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.228716 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5de80d59-44e0-48d2-88de-e51fdc1bd19e-cnibin\") pod \"multus-additional-cni-plugins-ml7g5\" (UID: \"5de80d59-44e0-48d2-88de-e51fdc1bd19e\") " pod="openshift-multus/multus-additional-cni-plugins-ml7g5" Sep 05 06:26:18.228907 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.228743 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-run-systemd\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.228907 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.228765 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/000daeb0-4f4d-497e-a81e-f7457115f0e9-ovn-node-metrics-cert\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.228907 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.228787 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-host-run-k8s-cni-cncf-io\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.228907 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.228811 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60b02f38-b310-4273-bc86-e3eb032c1cd4-host\") pod \"node-ca-m7rvs\" (UID: \"60b02f38-b310-4273-bc86-e3eb032c1cd4\") " pod="openshift-image-registry/node-ca-m7rvs" Sep 05 06:26:18.228907 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.228835 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb143a2d-2aaa-41ee-b476-2a03fa70ea4f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qw864\" (UID: \"fb143a2d-2aaa-41ee-b476-2a03fa70ea4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" Sep 05 06:26:18.228907 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.228862 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a3094f3-d739-47e4-b8a0-49335c888b12-dbus\") pod \"global-pull-secret-syncer-qzpdk\" (UID: \"8a3094f3-d739-47e4-b8a0-49335c888b12\") " pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:18.228907 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.228886 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-var-lib-openvswitch\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.229143 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.228910 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt4sx\" (UniqueName: \"kubernetes.io/projected/60296196-88ed-4a44-a893-d7715babb5f8-kube-api-access-vt4sx\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.229143 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.228932 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb143a2d-2aaa-41ee-b476-2a03fa70ea4f-socket-dir\") pod \"aws-ebs-csi-driver-node-qw864\" (UID: \"fb143a2d-2aaa-41ee-b476-2a03fa70ea4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" Sep 05 06:26:18.229143 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.228954 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-multus-cni-dir\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.229143 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.228977 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ch4c\" (UniqueName: \"kubernetes.io/projected/88f016f8-99a0-4560-b4e2-2abd92c5cf90-kube-api-access-8ch4c\") pod \"network-check-target-nccpk\" (UID: \"88f016f8-99a0-4560-b4e2-2abd92c5cf90\") " pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:18.229143 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.228999 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/000daeb0-4f4d-497e-a81e-f7457115f0e9-ovnkube-config\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.229143 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229022 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-host-run-netns\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.229143 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229066 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb143a2d-2aaa-41ee-b476-2a03fa70ea4f-registration-dir\") pod \"aws-ebs-csi-driver-node-qw864\" (UID: \"fb143a2d-2aaa-41ee-b476-2a03fa70ea4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" Sep 05 06:26:18.229143 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229094 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb143a2d-2aaa-41ee-b476-2a03fa70ea4f-device-dir\") pod \"aws-ebs-csi-driver-node-qw864\" (UID: \"fb143a2d-2aaa-41ee-b476-2a03fa70ea4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" Sep 05 06:26:18.229143 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229117 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhftv\" (UniqueName: \"kubernetes.io/projected/fb143a2d-2aaa-41ee-b476-2a03fa70ea4f-kube-api-access-fhftv\") pod \"aws-ebs-csi-driver-node-qw864\" (UID: \"fb143a2d-2aaa-41ee-b476-2a03fa70ea4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" Sep 05 06:26:18.229508 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229162 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-systemd-units\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.229508 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229190 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.229508 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229228 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-var-lib-openvswitch\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.229508 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229232 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aee7de64-f537-45d5-b9f9-202d8ebc279c-agent-certs\") pod \"konnectivity-agent-zqj4q\" (UID: \"aee7de64-f537-45d5-b9f9-202d8ebc279c\") " pod="kube-system/konnectivity-agent-zqj4q" Sep 05 06:26:18.229508 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229256 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-host-run-netns\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.229508 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229284 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-multus-conf-dir\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.229508 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229307 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3094f3-d739-47e4-b8a0-49335c888b12-original-pull-secret\") pod \"global-pull-secret-syncer-qzpdk\" (UID: \"8a3094f3-d739-47e4-b8a0-49335c888b12\") " pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:18.229508 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229335 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2gz5\" (UniqueName: \"kubernetes.io/projected/60b02f38-b310-4273-bc86-e3eb032c1cd4-kube-api-access-g2gz5\") pod \"node-ca-m7rvs\" (UID: \"60b02f38-b310-4273-bc86-e3eb032c1cd4\") " pod="openshift-image-registry/node-ca-m7rvs" Sep 05 06:26:18.229508 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229362 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ksks\" (UniqueName: \"kubernetes.io/projected/5de80d59-44e0-48d2-88de-e51fdc1bd19e-kube-api-access-7ksks\") pod \"multus-additional-cni-plugins-ml7g5\" (UID: \"5de80d59-44e0-48d2-88de-e51fdc1bd19e\") " pod="openshift-multus/multus-additional-cni-plugins-ml7g5" Sep 05 06:26:18.229508 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229404 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aee7de64-f537-45d5-b9f9-202d8ebc279c-konnectivity-ca\") pod \"konnectivity-agent-zqj4q\" (UID: \"aee7de64-f537-45d5-b9f9-202d8ebc279c\") " pod="kube-system/konnectivity-agent-zqj4q" Sep 05 06:26:18.229508 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229431 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.229508 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229459 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvskd\" (UniqueName: \"kubernetes.io/projected/9103306e-ef91-4539-b914-8c3cdb23ea94-kube-api-access-bvskd\") pod \"network-metrics-daemon-545p9\" (UID: \"9103306e-ef91-4539-b914-8c3cdb23ea94\") " pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:18.229508 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229485 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/000daeb0-4f4d-497e-a81e-f7457115f0e9-env-overrides\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.230148 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229517 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5de80d59-44e0-48d2-88de-e51fdc1bd19e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ml7g5\" (UID: \"5de80d59-44e0-48d2-88de-e51fdc1bd19e\") " pod="openshift-multus/multus-additional-cni-plugins-ml7g5" Sep 05 06:26:18.230148 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229548 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-run-openvswitch\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.230148 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229567 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.230148 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229578 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-multus-socket-dir-parent\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.230148 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229606 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a3094f3-d739-47e4-b8a0-49335c888b12-kubelet-config\") pod \"global-pull-secret-syncer-qzpdk\" (UID: \"8a3094f3-d739-47e4-b8a0-49335c888b12\") " pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:18.230148 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229631 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/60b02f38-b310-4273-bc86-e3eb032c1cd4-serviceca\") pod \"node-ca-m7rvs\" (UID: \"60b02f38-b310-4273-bc86-e3eb032c1cd4\") " pod="openshift-image-registry/node-ca-m7rvs" Sep 05 06:26:18.230148 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229669 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5de80d59-44e0-48d2-88de-e51fdc1bd19e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ml7g5\" (UID: \"5de80d59-44e0-48d2-88de-e51fdc1bd19e\") " pod="openshift-multus/multus-additional-cni-plugins-ml7g5" Sep 05 06:26:18.230148 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229714 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-system-cni-dir\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.230148 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229741 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-cnibin\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.230148 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229796 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-etc-kubernetes\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.230148 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229826 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5de80d59-44e0-48d2-88de-e51fdc1bd19e-system-cni-dir\") pod \"multus-additional-cni-plugins-ml7g5\" (UID: \"5de80d59-44e0-48d2-88de-e51fdc1bd19e\") " pod="openshift-multus/multus-additional-cni-plugins-ml7g5" Sep 05 06:26:18.230148 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229855 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60296196-88ed-4a44-a893-d7715babb5f8-cni-binary-copy\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.230148 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229861 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60296196-88ed-4a44-a893-d7715babb5f8-multus-daemon-config\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.230148 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229903 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5de80d59-44e0-48d2-88de-e51fdc1bd19e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ml7g5\" (UID: \"5de80d59-44e0-48d2-88de-e51fdc1bd19e\") " pod="openshift-multus/multus-additional-cni-plugins-ml7g5" Sep 05 06:26:18.230148 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229935 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5de80d59-44e0-48d2-88de-e51fdc1bd19e-cnibin\") pod \"multus-additional-cni-plugins-ml7g5\" (UID: \"5de80d59-44e0-48d2-88de-e51fdc1bd19e\") " pod="openshift-multus/multus-additional-cni-plugins-ml7g5" Sep 05 06:26:18.230148 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229937 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-host-slash\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.230148 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229980 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-host-slash\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.231053 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.230018 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-log-socket\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.231053 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.230027 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60b02f38-b310-4273-bc86-e3eb032c1cd4-host\") pod \"node-ca-m7rvs\" (UID: \"60b02f38-b310-4273-bc86-e3eb032c1cd4\") " pod="openshift-image-registry/node-ca-m7rvs" Sep 05 06:26:18.231053 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.230046 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-host-cni-bin\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.231053 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.230048 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-multus-cni-dir\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.231053 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.230062 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/000daeb0-4f4d-497e-a81e-f7457115f0e9-ovnkube-config\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.231053 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.230073 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-host-cni-netd\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.231053 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.230080 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-host-run-k8s-cni-cncf-io\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.231053 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.230140 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-host-run-netns\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.231053 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.230168 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb143a2d-2aaa-41ee-b476-2a03fa70ea4f-socket-dir\") pod \"aws-ebs-csi-driver-node-qw864\" (UID: \"fb143a2d-2aaa-41ee-b476-2a03fa70ea4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" Sep 05 06:26:18.231053 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.230182 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.231053 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.230195 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-system-cni-dir\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.231053 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.229517 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-multus-conf-dir\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.231053 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.230229 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-run-systemd\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.231053 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.230233 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-host-var-lib-cni-multus\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.231053 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.230267 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-host-run-netns\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.231053 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.230274 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb143a2d-2aaa-41ee-b476-2a03fa70ea4f-registration-dir\") pod \"aws-ebs-csi-driver-node-qw864\" (UID: \"fb143a2d-2aaa-41ee-b476-2a03fa70ea4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" Sep 05 06:26:18.231053 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.230316 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-host-var-lib-kubelet\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.231053 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.230324 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-cnibin\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.232050 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.230362 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5de80d59-44e0-48d2-88de-e51fdc1bd19e-cni-binary-copy\") pod \"multus-additional-cni-plugins-ml7g5\" (UID: \"5de80d59-44e0-48d2-88de-e51fdc1bd19e\") " pod="openshift-multus/multus-additional-cni-plugins-ml7g5" Sep 05 06:26:18.232050 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.230404 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-run-openvswitch\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.232050 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.230413 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs\") pod \"network-metrics-daemon-545p9\" (UID: \"9103306e-ef91-4539-b914-8c3cdb23ea94\") " pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:18.232050 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.230446 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-host-run-multus-certs\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.232050 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.230484 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-etc-openvswitch\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.232050 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.230446 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-etc-kubernetes\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.232050 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.230524 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-systemd-units\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.232744 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.232637 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-multus-socket-dir-parent\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.233356 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.233203 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5de80d59-44e0-48d2-88de-e51fdc1bd19e-cni-binary-copy\") pod \"multus-additional-cni-plugins-ml7g5\" (UID: \"5de80d59-44e0-48d2-88de-e51fdc1bd19e\") " pod="openshift-multus/multus-additional-cni-plugins-ml7g5" Sep 05 06:26:18.233742 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.233695 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5de80d59-44e0-48d2-88de-e51fdc1bd19e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ml7g5\" (UID: \"5de80d59-44e0-48d2-88de-e51fdc1bd19e\") " pod="openshift-multus/multus-additional-cni-plugins-ml7g5" Sep 05 06:26:18.233834 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.233762 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-host-var-lib-kubelet\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.233834 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.233829 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a3094f3-d739-47e4-b8a0-49335c888b12-kubelet-config\") pod \"global-pull-secret-syncer-qzpdk\" (UID: \"8a3094f3-d739-47e4-b8a0-49335c888b12\") " pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:18.234261 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.234234 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/60b02f38-b310-4273-bc86-e3eb032c1cd4-serviceca\") pod \"node-ca-m7rvs\" (UID: \"60b02f38-b310-4273-bc86-e3eb032c1cd4\") " pod="openshift-image-registry/node-ca-m7rvs" Sep 05 06:26:18.234445 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:18.234339 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 05 06:26:18.234445 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:18.234439 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a3094f3-d739-47e4-b8a0-49335c888b12-original-pull-secret podName:8a3094f3-d739-47e4-b8a0-49335c888b12 nodeName:}" failed. No retries permitted until 2026-09-05 06:26:18.734416705 +0000 UTC m=+3.118959831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a3094f3-d739-47e4-b8a0-49335c888b12-original-pull-secret") pod "global-pull-secret-syncer-qzpdk" (UID: "8a3094f3-d739-47e4-b8a0-49335c888b12") : object "kube-system"/"original-pull-secret" not registered Sep 05 06:26:18.236930 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.234920 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-host-cni-bin\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.236930 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.235107 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/000daeb0-4f4d-497e-a81e-f7457115f0e9-env-overrides\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.236930 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.230412 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5de80d59-44e0-48d2-88de-e51fdc1bd19e-system-cni-dir\") pod \"multus-additional-cni-plugins-ml7g5\" (UID: \"5de80d59-44e0-48d2-88de-e51fdc1bd19e\") " pod="openshift-multus/multus-additional-cni-plugins-ml7g5" Sep 05 06:26:18.236930 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.235172 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aee7de64-f537-45d5-b9f9-202d8ebc279c-agent-certs\") pod \"konnectivity-agent-zqj4q\" (UID: \"aee7de64-f537-45d5-b9f9-202d8ebc279c\") " pod="kube-system/konnectivity-agent-zqj4q" Sep 05 06:26:18.236930 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.235286 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-host-cni-netd\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.236930 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.235325 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5de80d59-44e0-48d2-88de-e51fdc1bd19e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ml7g5\" (UID: \"5de80d59-44e0-48d2-88de-e51fdc1bd19e\") " pod="openshift-multus/multus-additional-cni-plugins-ml7g5" Sep 05 06:26:18.236930 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.235367 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb143a2d-2aaa-41ee-b476-2a03fa70ea4f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qw864\" (UID: \"fb143a2d-2aaa-41ee-b476-2a03fa70ea4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" Sep 05 06:26:18.236930 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:18.235490 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 06:26:18.236930 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.235499 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60296196-88ed-4a44-a893-d7715babb5f8-multus-daemon-config\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.236930 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:18.235702 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs podName:9103306e-ef91-4539-b914-8c3cdb23ea94 nodeName:}" failed. No retries permitted until 2026-09-05 06:26:18.735681619 +0000 UTC m=+3.120224740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs") pod "network-metrics-daemon-545p9" (UID: "9103306e-ef91-4539-b914-8c3cdb23ea94") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 06:26:18.236930 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.235802 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb143a2d-2aaa-41ee-b476-2a03fa70ea4f-device-dir\") pod \"aws-ebs-csi-driver-node-qw864\" (UID: \"fb143a2d-2aaa-41ee-b476-2a03fa70ea4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" Sep 05 06:26:18.236930 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.236170 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-log-socket\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.236930 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.236236 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a3094f3-d739-47e4-b8a0-49335c888b12-dbus\") pod \"global-pull-secret-syncer-qzpdk\" (UID: \"8a3094f3-d739-47e4-b8a0-49335c888b12\") " pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:18.236930 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.236306 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-run-ovn\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.236930 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.236402 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-etc-openvswitch\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.236930 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.236637 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aee7de64-f537-45d5-b9f9-202d8ebc279c-konnectivity-ca\") pod \"konnectivity-agent-zqj4q\" (UID: \"aee7de64-f537-45d5-b9f9-202d8ebc279c\") " pod="kube-system/konnectivity-agent-zqj4q" Sep 05 06:26:18.236930 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.236683 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-run-ovn\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.240716 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.236729 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/000daeb0-4f4d-497e-a81e-f7457115f0e9-ovnkube-script-lib\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.240716 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.236791 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb143a2d-2aaa-41ee-b476-2a03fa70ea4f-sys-fs\") pod \"aws-ebs-csi-driver-node-qw864\" (UID: \"fb143a2d-2aaa-41ee-b476-2a03fa70ea4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" Sep 05 06:26:18.240716 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.236852 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-host-run-multus-certs\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.240716 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.237076 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-host-var-lib-cni-bin\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.240716 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.237119 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-hostroot\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.240716 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.237052 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5de80d59-44e0-48d2-88de-e51fdc1bd19e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ml7g5\" (UID: \"5de80d59-44e0-48d2-88de-e51fdc1bd19e\") " pod="openshift-multus/multus-additional-cni-plugins-ml7g5" Sep 05 06:26:18.240716 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.237266 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb143a2d-2aaa-41ee-b476-2a03fa70ea4f-etc-selinux\") pod \"aws-ebs-csi-driver-node-qw864\" (UID: \"fb143a2d-2aaa-41ee-b476-2a03fa70ea4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" Sep 05 06:26:18.240716 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.237303 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-host-kubelet\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.240716 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.237333 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-node-log\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.240716 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.237525 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l87jb\" (UniqueName: \"kubernetes.io/projected/000daeb0-4f4d-497e-a81e-f7457115f0e9-kube-api-access-l87jb\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.240716 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.237567 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-host-kubelet\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.240716 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.237632 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb143a2d-2aaa-41ee-b476-2a03fa70ea4f-sys-fs\") pod \"aws-ebs-csi-driver-node-qw864\" (UID: \"fb143a2d-2aaa-41ee-b476-2a03fa70ea4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" Sep 05 06:26:18.240716 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.237640 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-hostroot\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.240716 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.237678 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60296196-88ed-4a44-a893-d7715babb5f8-host-var-lib-cni-bin\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.240716 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.237737 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb143a2d-2aaa-41ee-b476-2a03fa70ea4f-etc-selinux\") pod \"aws-ebs-csi-driver-node-qw864\" (UID: \"fb143a2d-2aaa-41ee-b476-2a03fa70ea4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" Sep 05 06:26:18.240716 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.237768 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/000daeb0-4f4d-497e-a81e-f7457115f0e9-ovnkube-script-lib\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.240716 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.237773 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/000daeb0-4f4d-497e-a81e-f7457115f0e9-node-log\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.241464 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.237673 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/000daeb0-4f4d-497e-a81e-f7457115f0e9-ovn-node-metrics-cert\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.242562 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:18.242542 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 05 06:26:18.242751 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:18.242736 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 05 06:26:18.242859 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.242780 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 05 06:26:18.244201 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:18.242855 2565 projected.go:196] Error preparing data for projected volume kube-api-access-8ch4c for pod openshift-network-diagnostics/network-check-target-nccpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 06:26:18.244201 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:18.243775 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88f016f8-99a0-4560-b4e2-2abd92c5cf90-kube-api-access-8ch4c podName:88f016f8-99a0-4560-b4e2-2abd92c5cf90 nodeName:}" failed. No retries permitted until 2026-09-05 06:26:18.743757794 +0000 UTC m=+3.128300919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8ch4c" (UniqueName: "kubernetes.io/projected/88f016f8-99a0-4560-b4e2-2abd92c5cf90-kube-api-access-8ch4c") pod "network-check-target-nccpk" (UID: "88f016f8-99a0-4560-b4e2-2abd92c5cf90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 06:26:18.245220 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.245199 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt4sx\" (UniqueName: \"kubernetes.io/projected/60296196-88ed-4a44-a893-d7715babb5f8-kube-api-access-vt4sx\") pod \"multus-xg8xw\" (UID: \"60296196-88ed-4a44-a893-d7715babb5f8\") " pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.247474 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.247429 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvskd\" (UniqueName: \"kubernetes.io/projected/9103306e-ef91-4539-b914-8c3cdb23ea94-kube-api-access-bvskd\") pod \"network-metrics-daemon-545p9\" (UID: \"9103306e-ef91-4539-b914-8c3cdb23ea94\") " pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:18.255125 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.255083 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2gz5\" (UniqueName: \"kubernetes.io/projected/60b02f38-b310-4273-bc86-e3eb032c1cd4-kube-api-access-g2gz5\") pod \"node-ca-m7rvs\" (UID: \"60b02f38-b310-4273-bc86-e3eb032c1cd4\") " pod="openshift-image-registry/node-ca-m7rvs" Sep 05 06:26:18.255875 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.255855 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l87jb\" (UniqueName: \"kubernetes.io/projected/000daeb0-4f4d-497e-a81e-f7457115f0e9-kube-api-access-l87jb\") pod \"ovnkube-node-6mvpl\" (UID: \"000daeb0-4f4d-497e-a81e-f7457115f0e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.257455 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.257434 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhftv\" (UniqueName: \"kubernetes.io/projected/fb143a2d-2aaa-41ee-b476-2a03fa70ea4f-kube-api-access-fhftv\") pod \"aws-ebs-csi-driver-node-qw864\" (UID: \"fb143a2d-2aaa-41ee-b476-2a03fa70ea4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" Sep 05 06:26:18.260191 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.260163 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ksks\" (UniqueName: \"kubernetes.io/projected/5de80d59-44e0-48d2-88de-e51fdc1bd19e-kube-api-access-7ksks\") pod \"multus-additional-cni-plugins-ml7g5\" (UID: \"5de80d59-44e0-48d2-88de-e51fdc1bd19e\") " pod="openshift-multus/multus-additional-cni-plugins-ml7g5" Sep 05 06:26:18.327987 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.327948 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gf7px" Sep 05 06:26:18.338436 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.338411 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dzlmk" Sep 05 06:26:18.351078 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.350876 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m7rvs" Sep 05 06:26:18.359292 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.359127 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" Sep 05 06:26:18.367143 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.367116 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xg8xw" Sep 05 06:26:18.377290 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.377264 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ml7g5" Sep 05 06:26:18.382287 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.382261 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zqj4q" Sep 05 06:26:18.394971 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.394944 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:18.401045 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.401021 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" Sep 05 06:26:18.742771 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.742683 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs\") pod \"network-metrics-daemon-545p9\" (UID: \"9103306e-ef91-4539-b914-8c3cdb23ea94\") " pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:18.742771 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.742766 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3094f3-d739-47e4-b8a0-49335c888b12-original-pull-secret\") pod \"global-pull-secret-syncer-qzpdk\" (UID: \"8a3094f3-d739-47e4-b8a0-49335c888b12\") " pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:18.743001 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:18.742861 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 05 06:26:18.743001 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:18.742863 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 06:26:18.743001 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:18.742929 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a3094f3-d739-47e4-b8a0-49335c888b12-original-pull-secret podName:8a3094f3-d739-47e4-b8a0-49335c888b12 nodeName:}" failed. No retries permitted until 2026-09-05 06:26:19.742907831 +0000 UTC m=+4.127450954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a3094f3-d739-47e4-b8a0-49335c888b12-original-pull-secret") pod "global-pull-secret-syncer-qzpdk" (UID: "8a3094f3-d739-47e4-b8a0-49335c888b12") : object "kube-system"/"original-pull-secret" not registered Sep 05 06:26:18.743001 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:18.742947 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs podName:9103306e-ef91-4539-b914-8c3cdb23ea94 nodeName:}" failed. No retries permitted until 2026-09-05 06:26:19.742939149 +0000 UTC m=+4.127482253 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs") pod "network-metrics-daemon-545p9" (UID: "9103306e-ef91-4539-b914-8c3cdb23ea94") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 06:26:18.843747 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:18.843700 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ch4c\" (UniqueName: \"kubernetes.io/projected/88f016f8-99a0-4560-b4e2-2abd92c5cf90-kube-api-access-8ch4c\") pod \"network-check-target-nccpk\" (UID: \"88f016f8-99a0-4560-b4e2-2abd92c5cf90\") " pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:18.843927 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:18.843908 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 05 06:26:18.843999 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:18.843936 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 05 06:26:18.843999 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:18.843947 2565 projected.go:196] Error preparing data for projected volume kube-api-access-8ch4c for pod openshift-network-diagnostics/network-check-target-nccpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 06:26:18.844072 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:18.844010 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88f016f8-99a0-4560-b4e2-2abd92c5cf90-kube-api-access-8ch4c podName:88f016f8-99a0-4560-b4e2-2abd92c5cf90 nodeName:}" failed. No retries permitted until 2026-09-05 06:26:19.843995736 +0000 UTC m=+4.228538843 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8ch4c" (UniqueName: "kubernetes.io/projected/88f016f8-99a0-4560-b4e2-2abd92c5cf90-kube-api-access-8ch4c") pod "network-check-target-nccpk" (UID: "88f016f8-99a0-4560-b4e2-2abd92c5cf90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 06:26:19.064247 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:19.063989 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 06:21:18 +0000 UTC" deadline="2028-02-16 19:47:35.702700164 +0000 UTC" Sep 05 06:26:19.064247 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:19.064194 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12709h21m16.638510067s" Sep 05 06:26:19.159325 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:19.159277 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:19.159493 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:19.159419 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-545p9" podUID="9103306e-ef91-4539-b914-8c3cdb23ea94" Sep 05 06:26:19.340407 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:19.340370 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee7de64_f537_45d5_b9f9_202d8ebc279c.slice/crio-b6ff20b6b4a79ce39e7dae697e1cb47af9b22e6091d342de1e674a3f2708524a WatchSource:0}: Error finding container b6ff20b6b4a79ce39e7dae697e1cb47af9b22e6091d342de1e674a3f2708524a: Status 404 returned error can't find the container with id b6ff20b6b4a79ce39e7dae697e1cb47af9b22e6091d342de1e674a3f2708524a Sep 05 06:26:19.341912 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:19.341883 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59539926_4278_4795_8ff8_f3482b69df83.slice/crio-e58838bcca9f69f41e8d4bef6cd0c2304ee5f209bc0ce5d2f653c41e71048f88 WatchSource:0}: Error finding container e58838bcca9f69f41e8d4bef6cd0c2304ee5f209bc0ce5d2f653c41e71048f88: Status 404 returned error can't find the container with id e58838bcca9f69f41e8d4bef6cd0c2304ee5f209bc0ce5d2f653c41e71048f88 Sep 05 06:26:19.343277 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:19.343248 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de80d59_44e0_48d2_88de_e51fdc1bd19e.slice/crio-3c5253e8c49f38c96cc9de52b148d3e2a9aa3027910bce72f83a7f83fe88d300 WatchSource:0}: Error finding container 3c5253e8c49f38c96cc9de52b148d3e2a9aa3027910bce72f83a7f83fe88d300: Status 404 returned error can't find the container with id 3c5253e8c49f38c96cc9de52b148d3e2a9aa3027910bce72f83a7f83fe88d300 Sep 05 06:26:19.344781 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:19.344614 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1707699_6ecc_4982_9a8e_b1e22f0e751a.slice/crio-bd3fc91c0d0eaed3380b30cc9543902caf203bfd88659dde856fbc90f130ccf9 WatchSource:0}: Error finding container bd3fc91c0d0eaed3380b30cc9543902caf203bfd88659dde856fbc90f130ccf9: Status 404 returned error can't find the container with id bd3fc91c0d0eaed3380b30cc9543902caf203bfd88659dde856fbc90f130ccf9 Sep 05 06:26:19.350984 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:19.350957 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb143a2d_2aaa_41ee_b476_2a03fa70ea4f.slice/crio-72101ef68305d3814a7d17aec3b4c96ab95134b74e92b47b134c170e2708db4c WatchSource:0}: Error finding container 72101ef68305d3814a7d17aec3b4c96ab95134b74e92b47b134c170e2708db4c: Status 404 returned error can't find the container with id 72101ef68305d3814a7d17aec3b4c96ab95134b74e92b47b134c170e2708db4c Sep 05 06:26:19.354446 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:19.354384 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d3234f_c67d_4d69_9c79_e0f46c361ae0.slice/crio-471f5411f43aadf3b0668a377549052fd2a6f84f0c2588b4471aeba5bae29b40 WatchSource:0}: Error finding container 471f5411f43aadf3b0668a377549052fd2a6f84f0c2588b4471aeba5bae29b40: Status 404 returned error can't find the container with id 471f5411f43aadf3b0668a377549052fd2a6f84f0c2588b4471aeba5bae29b40 Sep 05 06:26:19.354906 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:19.354879 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b02f38_b310_4273_bc86_e3eb032c1cd4.slice/crio-4e66e1beeaa86c4ffcba91ca34f62114f8cadaf33549865db5290167da6bcc14 WatchSource:0}: Error finding container 4e66e1beeaa86c4ffcba91ca34f62114f8cadaf33549865db5290167da6bcc14: Status 404 returned error can't find the container with id 4e66e1beeaa86c4ffcba91ca34f62114f8cadaf33549865db5290167da6bcc14 Sep 05 06:26:19.355131 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:19.355109 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod000daeb0_4f4d_497e_a81e_f7457115f0e9.slice/crio-553a62071ea6bb62125d13aefee93390957ae6ebd3673d512d75fb550a5cc238 WatchSource:0}: Error finding container 553a62071ea6bb62125d13aefee93390957ae6ebd3673d512d75fb550a5cc238: Status 404 returned error can't find the container with id 553a62071ea6bb62125d13aefee93390957ae6ebd3673d512d75fb550a5cc238 Sep 05 06:26:19.357224 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:19.357202 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60296196_88ed_4a44_a893_d7715babb5f8.slice/crio-9f0b36d073a8f3cd7bc7c03aaab21bd7c998116dbf6f2ae619c03b4a85fb1863 WatchSource:0}: Error finding container 9f0b36d073a8f3cd7bc7c03aaab21bd7c998116dbf6f2ae619c03b4a85fb1863: Status 404 returned error can't find the container with id 9f0b36d073a8f3cd7bc7c03aaab21bd7c998116dbf6f2ae619c03b4a85fb1863 Sep 05 06:26:19.367322 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:19.367281 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 05 06:26:19.728026 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:19.727999 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 05 06:26:19.751239 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:19.751200 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3094f3-d739-47e4-b8a0-49335c888b12-original-pull-secret\") pod \"global-pull-secret-syncer-qzpdk\" (UID: \"8a3094f3-d739-47e4-b8a0-49335c888b12\") " pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:19.751435 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:19.751275 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs\") pod \"network-metrics-daemon-545p9\" (UID: \"9103306e-ef91-4539-b914-8c3cdb23ea94\") " pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:19.751435 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:19.751399 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 05 06:26:19.751435 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:19.751415 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 06:26:19.751615 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:19.751488 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a3094f3-d739-47e4-b8a0-49335c888b12-original-pull-secret podName:8a3094f3-d739-47e4-b8a0-49335c888b12 nodeName:}" failed. No retries permitted until 2026-09-05 06:26:21.751464387 +0000 UTC m=+6.136007516 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a3094f3-d739-47e4-b8a0-49335c888b12-original-pull-secret") pod "global-pull-secret-syncer-qzpdk" (UID: "8a3094f3-d739-47e4-b8a0-49335c888b12") : object "kube-system"/"original-pull-secret" not registered Sep 05 06:26:19.751615 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:19.751513 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs podName:9103306e-ef91-4539-b914-8c3cdb23ea94 nodeName:}" failed. No retries permitted until 2026-09-05 06:26:21.75150183 +0000 UTC m=+6.136044948 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs") pod "network-metrics-daemon-545p9" (UID: "9103306e-ef91-4539-b914-8c3cdb23ea94") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 06:26:19.851946 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:19.851910 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ch4c\" (UniqueName: \"kubernetes.io/projected/88f016f8-99a0-4560-b4e2-2abd92c5cf90-kube-api-access-8ch4c\") pod \"network-check-target-nccpk\" (UID: \"88f016f8-99a0-4560-b4e2-2abd92c5cf90\") " pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:19.852183 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:19.852117 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 05 06:26:19.852183 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:19.852154 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 05 06:26:19.852183 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:19.852168 2565 projected.go:196] Error preparing data for projected volume kube-api-access-8ch4c for pod openshift-network-diagnostics/network-check-target-nccpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 06:26:19.852460 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:19.852246 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88f016f8-99a0-4560-b4e2-2abd92c5cf90-kube-api-access-8ch4c podName:88f016f8-99a0-4560-b4e2-2abd92c5cf90 nodeName:}" failed. No retries permitted until 2026-09-05 06:26:21.852223597 +0000 UTC m=+6.236766714 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8ch4c" (UniqueName: "kubernetes.io/projected/88f016f8-99a0-4560-b4e2-2abd92c5cf90-kube-api-access-8ch4c") pod "network-check-target-nccpk" (UID: "88f016f8-99a0-4560-b4e2-2abd92c5cf90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 06:26:20.064863 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:20.064764 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 06:21:18 +0000 UTC" deadline="2028-02-02 11:35:02.517855822 +0000 UTC" Sep 05 06:26:20.064863 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:20.064813 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12365h8m42.45304854s" Sep 05 06:26:20.160908 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:20.160165 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:20.160908 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:20.160309 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nccpk" podUID="88f016f8-99a0-4560-b4e2-2abd92c5cf90" Sep 05 06:26:20.160908 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:20.160735 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:20.160908 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:20.160852 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzpdk" podUID="8a3094f3-d739-47e4-b8a0-49335c888b12" Sep 05 06:26:20.191415 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:20.191103 2565 generic.go:356] "Generic (PLEG): container finished" podID="0d39c298c5184aab44649905d0fcf62a" containerID="88cf525674535e2a8728fce5d9f5e4c2997a9327ee352c6ce711e7d67c8a5190" exitCode=0 Sep 05 06:26:20.195415 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:20.192221 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" event={"ID":"0d39c298c5184aab44649905d0fcf62a","Type":"ContainerDied","Data":"88cf525674535e2a8728fce5d9f5e4c2997a9327ee352c6ce711e7d67c8a5190"} Sep 05 06:26:20.205954 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:20.205921 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal" event={"ID":"6a158339afd761bc08604ebeb5aa4b2a","Type":"ContainerStarted","Data":"bf47242b23172cb948843a6976165da3400f14e856af9008ce6d3358ab0beb9f"} Sep 05 06:26:20.211048 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:20.210992 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xg8xw" event={"ID":"60296196-88ed-4a44-a893-d7715babb5f8","Type":"ContainerStarted","Data":"9f0b36d073a8f3cd7bc7c03aaab21bd7c998116dbf6f2ae619c03b4a85fb1863"} Sep 05 06:26:20.214565 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:20.214534 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" event={"ID":"000daeb0-4f4d-497e-a81e-f7457115f0e9","Type":"ContainerStarted","Data":"553a62071ea6bb62125d13aefee93390957ae6ebd3673d512d75fb550a5cc238"} Sep 05 06:26:20.226155 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:20.226085 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m7rvs" event={"ID":"60b02f38-b310-4273-bc86-e3eb032c1cd4","Type":"ContainerStarted","Data":"4e66e1beeaa86c4ffcba91ca34f62114f8cadaf33549865db5290167da6bcc14"} Sep 05 06:26:20.230519 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:20.230485 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" event={"ID":"e8d3234f-c67d-4d69-9c79-e0f46c361ae0","Type":"ContainerStarted","Data":"471f5411f43aadf3b0668a377549052fd2a6f84f0c2588b4471aeba5bae29b40"} Sep 05 06:26:20.244354 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:20.244314 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ml7g5" event={"ID":"5de80d59-44e0-48d2-88de-e51fdc1bd19e","Type":"ContainerStarted","Data":"3c5253e8c49f38c96cc9de52b148d3e2a9aa3027910bce72f83a7f83fe88d300"} Sep 05 06:26:20.255793 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:20.255698 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zqj4q" event={"ID":"aee7de64-f537-45d5-b9f9-202d8ebc279c","Type":"ContainerStarted","Data":"b6ff20b6b4a79ce39e7dae697e1cb47af9b22e6091d342de1e674a3f2708524a"} Sep 05 06:26:20.271171 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:20.271044 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" event={"ID":"fb143a2d-2aaa-41ee-b476-2a03fa70ea4f","Type":"ContainerStarted","Data":"72101ef68305d3814a7d17aec3b4c96ab95134b74e92b47b134c170e2708db4c"} Sep 05 06:26:20.292421 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:20.292268 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gf7px" event={"ID":"a1707699-6ecc-4982-9a8e-b1e22f0e751a","Type":"ContainerStarted","Data":"bd3fc91c0d0eaed3380b30cc9543902caf203bfd88659dde856fbc90f130ccf9"} Sep 05 06:26:20.314405 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:20.314348 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dzlmk" event={"ID":"59539926-4278-4795-8ff8-f3482b69df83","Type":"ContainerStarted","Data":"e58838bcca9f69f41e8d4bef6cd0c2304ee5f209bc0ce5d2f653c41e71048f88"} Sep 05 06:26:21.160690 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:21.160109 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:21.160690 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:21.160271 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-545p9" podUID="9103306e-ef91-4539-b914-8c3cdb23ea94" Sep 05 06:26:21.335701 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:21.335589 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" event={"ID":"0d39c298c5184aab44649905d0fcf62a","Type":"ContainerStarted","Data":"6e97fedbb1726043fed49194b1a92cf59eaca60c527a69313ba878aff0a76801"} Sep 05 06:26:21.360228 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:21.359059 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal" podStartSLOduration=3.359032829 podStartE2EDuration="3.359032829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 06:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:26:20.234590203 +0000 UTC m=+4.619133333" watchObservedRunningTime="2026-09-05 06:26:21.359032829 +0000 UTC m=+5.743575960" Sep 05 06:26:21.769836 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:21.769790 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3094f3-d739-47e4-b8a0-49335c888b12-original-pull-secret\") pod \"global-pull-secret-syncer-qzpdk\" (UID: \"8a3094f3-d739-47e4-b8a0-49335c888b12\") " pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:21.770051 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:21.769871 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs\") pod \"network-metrics-daemon-545p9\" (UID: \"9103306e-ef91-4539-b914-8c3cdb23ea94\") " pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:21.770051 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:21.770003 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 06:26:21.770256 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:21.770070 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs podName:9103306e-ef91-4539-b914-8c3cdb23ea94 nodeName:}" failed. No retries permitted until 2026-09-05 06:26:25.770051102 +0000 UTC m=+10.154594224 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs") pod "network-metrics-daemon-545p9" (UID: "9103306e-ef91-4539-b914-8c3cdb23ea94") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 06:26:21.770256 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:21.770144 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 05 06:26:21.770256 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:21.770239 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a3094f3-d739-47e4-b8a0-49335c888b12-original-pull-secret podName:8a3094f3-d739-47e4-b8a0-49335c888b12 nodeName:}" failed. No retries permitted until 2026-09-05 06:26:25.77016974 +0000 UTC m=+10.154712860 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a3094f3-d739-47e4-b8a0-49335c888b12-original-pull-secret") pod "global-pull-secret-syncer-qzpdk" (UID: "8a3094f3-d739-47e4-b8a0-49335c888b12") : object "kube-system"/"original-pull-secret" not registered Sep 05 06:26:21.871033 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:21.870435 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ch4c\" (UniqueName: \"kubernetes.io/projected/88f016f8-99a0-4560-b4e2-2abd92c5cf90-kube-api-access-8ch4c\") pod \"network-check-target-nccpk\" (UID: \"88f016f8-99a0-4560-b4e2-2abd92c5cf90\") " pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:21.871033 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:21.870596 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 05 06:26:21.871033 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:21.870613 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 05 06:26:21.871033 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:21.870623 2565 projected.go:196] Error preparing data for projected volume kube-api-access-8ch4c for pod openshift-network-diagnostics/network-check-target-nccpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 06:26:21.871033 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:21.870675 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88f016f8-99a0-4560-b4e2-2abd92c5cf90-kube-api-access-8ch4c podName:88f016f8-99a0-4560-b4e2-2abd92c5cf90 nodeName:}" failed. No retries permitted until 2026-09-05 06:26:25.87066069 +0000 UTC m=+10.255203793 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8ch4c" (UniqueName: "kubernetes.io/projected/88f016f8-99a0-4560-b4e2-2abd92c5cf90-kube-api-access-8ch4c") pod "network-check-target-nccpk" (UID: "88f016f8-99a0-4560-b4e2-2abd92c5cf90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 06:26:22.160033 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:22.159483 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:22.160033 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:22.159622 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nccpk" podUID="88f016f8-99a0-4560-b4e2-2abd92c5cf90" Sep 05 06:26:22.160681 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:22.160461 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:22.160681 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:22.160587 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzpdk" podUID="8a3094f3-d739-47e4-b8a0-49335c888b12" Sep 05 06:26:23.160737 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:23.160113 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:23.160737 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:23.160284 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-545p9" podUID="9103306e-ef91-4539-b914-8c3cdb23ea94" Sep 05 06:26:24.161766 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:24.161734 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:24.162268 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:24.161862 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nccpk" podUID="88f016f8-99a0-4560-b4e2-2abd92c5cf90" Sep 05 06:26:24.162377 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:24.162338 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:24.162451 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:24.162429 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzpdk" podUID="8a3094f3-d739-47e4-b8a0-49335c888b12" Sep 05 06:26:25.159487 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:25.159450 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:25.159693 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:25.159605 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-545p9" podUID="9103306e-ef91-4539-b914-8c3cdb23ea94" Sep 05 06:26:25.803370 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:25.802727 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3094f3-d739-47e4-b8a0-49335c888b12-original-pull-secret\") pod \"global-pull-secret-syncer-qzpdk\" (UID: \"8a3094f3-d739-47e4-b8a0-49335c888b12\") " pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:25.803370 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:25.802804 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs\") pod \"network-metrics-daemon-545p9\" (UID: \"9103306e-ef91-4539-b914-8c3cdb23ea94\") " pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:25.803370 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:25.802945 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 06:26:25.803370 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:25.803009 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs podName:9103306e-ef91-4539-b914-8c3cdb23ea94 nodeName:}" failed. No retries permitted until 2026-09-05 06:26:33.80298964 +0000 UTC m=+18.187532744 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs") pod "network-metrics-daemon-545p9" (UID: "9103306e-ef91-4539-b914-8c3cdb23ea94") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 06:26:25.803958 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:25.803428 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 05 06:26:25.803958 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:25.803502 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a3094f3-d739-47e4-b8a0-49335c888b12-original-pull-secret podName:8a3094f3-d739-47e4-b8a0-49335c888b12 nodeName:}" failed. No retries permitted until 2026-09-05 06:26:33.803484367 +0000 UTC m=+18.188027483 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a3094f3-d739-47e4-b8a0-49335c888b12-original-pull-secret") pod "global-pull-secret-syncer-qzpdk" (UID: "8a3094f3-d739-47e4-b8a0-49335c888b12") : object "kube-system"/"original-pull-secret" not registered Sep 05 06:26:25.903426 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:25.903387 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ch4c\" (UniqueName: \"kubernetes.io/projected/88f016f8-99a0-4560-b4e2-2abd92c5cf90-kube-api-access-8ch4c\") pod \"network-check-target-nccpk\" (UID: \"88f016f8-99a0-4560-b4e2-2abd92c5cf90\") " pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:25.903612 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:25.903544 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 05 06:26:25.903612 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:25.903566 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 05 06:26:25.903612 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:25.903578 2565 projected.go:196] Error preparing data for projected volume kube-api-access-8ch4c for pod openshift-network-diagnostics/network-check-target-nccpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 06:26:25.903748 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:25.903639 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88f016f8-99a0-4560-b4e2-2abd92c5cf90-kube-api-access-8ch4c podName:88f016f8-99a0-4560-b4e2-2abd92c5cf90 nodeName:}" failed. No retries permitted until 2026-09-05 06:26:33.90362126 +0000 UTC m=+18.288164383 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8ch4c" (UniqueName: "kubernetes.io/projected/88f016f8-99a0-4560-b4e2-2abd92c5cf90-kube-api-access-8ch4c") pod "network-check-target-nccpk" (UID: "88f016f8-99a0-4560-b4e2-2abd92c5cf90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 06:26:26.161326 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:26.160850 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:26.161326 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:26.160957 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nccpk" podUID="88f016f8-99a0-4560-b4e2-2abd92c5cf90" Sep 05 06:26:26.161810 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:26.161621 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:26.161810 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:26.161728 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzpdk" podUID="8a3094f3-d739-47e4-b8a0-49335c888b12" Sep 05 06:26:27.159408 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:27.159372 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:27.159811 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:27.159518 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-545p9" podUID="9103306e-ef91-4539-b914-8c3cdb23ea94" Sep 05 06:26:28.161076 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:28.160425 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:28.161076 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:28.160553 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nccpk" podUID="88f016f8-99a0-4560-b4e2-2abd92c5cf90" Sep 05 06:26:28.161076 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:28.160932 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:28.161076 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:28.161030 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzpdk" podUID="8a3094f3-d739-47e4-b8a0-49335c888b12" Sep 05 06:26:29.159886 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:29.159848 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:29.160070 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:29.159996 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-545p9" podUID="9103306e-ef91-4539-b914-8c3cdb23ea94" Sep 05 06:26:30.160279 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:30.160230 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:30.160754 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:30.160400 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzpdk" podUID="8a3094f3-d739-47e4-b8a0-49335c888b12" Sep 05 06:26:30.160754 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:30.160451 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:30.160754 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:30.160567 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nccpk" podUID="88f016f8-99a0-4560-b4e2-2abd92c5cf90" Sep 05 06:26:31.159405 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:31.159369 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:31.159599 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:31.159505 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-545p9" podUID="9103306e-ef91-4539-b914-8c3cdb23ea94" Sep 05 06:26:32.161590 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:32.161556 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:32.162001 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:32.161687 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzpdk" podUID="8a3094f3-d739-47e4-b8a0-49335c888b12" Sep 05 06:26:32.162071 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:32.162057 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:32.162150 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:32.162125 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nccpk" podUID="88f016f8-99a0-4560-b4e2-2abd92c5cf90" Sep 05 06:26:33.159380 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:33.159339 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:33.159583 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:33.159472 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-545p9" podUID="9103306e-ef91-4539-b914-8c3cdb23ea94" Sep 05 06:26:33.864885 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:33.864837 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3094f3-d739-47e4-b8a0-49335c888b12-original-pull-secret\") pod \"global-pull-secret-syncer-qzpdk\" (UID: \"8a3094f3-d739-47e4-b8a0-49335c888b12\") " pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:33.865484 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:33.864900 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs\") pod \"network-metrics-daemon-545p9\" (UID: \"9103306e-ef91-4539-b914-8c3cdb23ea94\") " pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:33.865484 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:33.864998 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 05 06:26:33.865484 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:33.865007 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 06:26:33.865484 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:33.865068 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a3094f3-d739-47e4-b8a0-49335c888b12-original-pull-secret podName:8a3094f3-d739-47e4-b8a0-49335c888b12 nodeName:}" failed. No retries permitted until 2026-09-05 06:26:49.865052432 +0000 UTC m=+34.249595535 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a3094f3-d739-47e4-b8a0-49335c888b12-original-pull-secret") pod "global-pull-secret-syncer-qzpdk" (UID: "8a3094f3-d739-47e4-b8a0-49335c888b12") : object "kube-system"/"original-pull-secret" not registered Sep 05 06:26:33.865484 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:33.865088 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs podName:9103306e-ef91-4539-b914-8c3cdb23ea94 nodeName:}" failed. No retries permitted until 2026-09-05 06:26:49.865078655 +0000 UTC m=+34.249621757 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs") pod "network-metrics-daemon-545p9" (UID: "9103306e-ef91-4539-b914-8c3cdb23ea94") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 06:26:33.965954 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:33.965906 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ch4c\" (UniqueName: \"kubernetes.io/projected/88f016f8-99a0-4560-b4e2-2abd92c5cf90-kube-api-access-8ch4c\") pod \"network-check-target-nccpk\" (UID: \"88f016f8-99a0-4560-b4e2-2abd92c5cf90\") " pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:33.966134 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:33.966059 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 05 06:26:33.966134 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:33.966075 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 05 06:26:33.966134 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:33.966085 2565 projected.go:196] Error preparing data for projected volume kube-api-access-8ch4c for pod openshift-network-diagnostics/network-check-target-nccpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 06:26:33.966270 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:33.966143 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88f016f8-99a0-4560-b4e2-2abd92c5cf90-kube-api-access-8ch4c podName:88f016f8-99a0-4560-b4e2-2abd92c5cf90 nodeName:}" failed. No retries permitted until 2026-09-05 06:26:49.966124991 +0000 UTC m=+34.350668097 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8ch4c" (UniqueName: "kubernetes.io/projected/88f016f8-99a0-4560-b4e2-2abd92c5cf90-kube-api-access-8ch4c") pod "network-check-target-nccpk" (UID: "88f016f8-99a0-4560-b4e2-2abd92c5cf90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 06:26:34.159483 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:34.159394 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:34.159652 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:34.159548 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzpdk" podUID="8a3094f3-d739-47e4-b8a0-49335c888b12" Sep 05 06:26:34.159652 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:34.159572 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:34.159773 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:34.159669 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nccpk" podUID="88f016f8-99a0-4560-b4e2-2abd92c5cf90" Sep 05 06:26:35.159459 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:35.159419 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:35.159930 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:35.159566 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-545p9" podUID="9103306e-ef91-4539-b914-8c3cdb23ea94" Sep 05 06:26:36.160235 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:36.160188 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:36.160726 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:36.160302 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzpdk" podUID="8a3094f3-d739-47e4-b8a0-49335c888b12" Sep 05 06:26:36.160726 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:36.160357 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:36.160726 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:36.160454 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nccpk" podUID="88f016f8-99a0-4560-b4e2-2abd92c5cf90" Sep 05 06:26:37.160006 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:37.159970 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:37.160205 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:37.160115 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-545p9" podUID="9103306e-ef91-4539-b914-8c3cdb23ea94" Sep 05 06:26:38.160366 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:38.160302 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:38.160746 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:38.160472 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nccpk" podUID="88f016f8-99a0-4560-b4e2-2abd92c5cf90" Sep 05 06:26:38.160880 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:38.160244 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:38.161002 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:38.160978 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzpdk" podUID="8a3094f3-d739-47e4-b8a0-49335c888b12" Sep 05 06:26:38.377037 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:38.376675 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xg8xw" event={"ID":"60296196-88ed-4a44-a893-d7715babb5f8","Type":"ContainerStarted","Data":"92b8ac5fb38be0d5c54c21f64b6adfd554685b338b2c7ed25f19afa53f0f7089"} Sep 05 06:26:38.378875 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:38.378736 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" event={"ID":"000daeb0-4f4d-497e-a81e-f7457115f0e9","Type":"ContainerStarted","Data":"94f948ef313009f85d8914f9f0cf0fe0c974e46567099bb4a8f9be463dea82ca"} Sep 05 06:26:38.378875 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:38.378806 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" event={"ID":"000daeb0-4f4d-497e-a81e-f7457115f0e9","Type":"ContainerStarted","Data":"d326a52622a2d816db12e32a30461d5247da4b378816c88a6fb44c47028abbdf"} Sep 05 06:26:38.378875 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:38.378858 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" event={"ID":"000daeb0-4f4d-497e-a81e-f7457115f0e9","Type":"ContainerStarted","Data":"c668e535138f4eaa0bd29a3d6227316025bf4268d56c5d1ed00c9c42484f9cce"} Sep 05 06:26:38.380354 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:38.380330 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m7rvs" event={"ID":"60b02f38-b310-4273-bc86-e3eb032c1cd4","Type":"ContainerStarted","Data":"c4c00d1cbd987a01e556f27921dc8da4f32e1a4ff0b54d072303d96ee9a18858"} Sep 05 06:26:38.381770 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:38.381751 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" event={"ID":"e8d3234f-c67d-4d69-9c79-e0f46c361ae0","Type":"ContainerStarted","Data":"7f2bc41f50c4c5b1490ff5d199562599bcecd691e077447dfee112d1c7449d59"} Sep 05 06:26:38.383290 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:38.383270 2565 generic.go:356] "Generic (PLEG): container finished" podID="5de80d59-44e0-48d2-88de-e51fdc1bd19e" containerID="f3b8272e5ac8f570000bd46dca68233421c4dd0f5d923ee505449d15507c05bb" exitCode=0 Sep 05 06:26:38.383406 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:38.383358 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ml7g5" event={"ID":"5de80d59-44e0-48d2-88de-e51fdc1bd19e","Type":"ContainerDied","Data":"f3b8272e5ac8f570000bd46dca68233421c4dd0f5d923ee505449d15507c05bb"} Sep 05 06:26:38.384988 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:38.384880 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zqj4q" event={"ID":"aee7de64-f537-45d5-b9f9-202d8ebc279c","Type":"ContainerStarted","Data":"845fdbcad1e332eaed1ea0d1906d2ebd71d6febf1b1f46c4800f215a893b65fe"} Sep 05 06:26:38.386818 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:38.386800 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" event={"ID":"fb143a2d-2aaa-41ee-b476-2a03fa70ea4f","Type":"ContainerStarted","Data":"146aad02a441438566d81761af21eda2fe917bd6bd93ad8ba0d24854fd2400bb"} Sep 05 06:26:38.388229 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:38.388210 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dzlmk" event={"ID":"59539926-4278-4795-8ff8-f3482b69df83","Type":"ContainerStarted","Data":"aacd9749034974c1dfa54eb1b2a1850be4ef351e5522ea5d1b43e4ce45c54b88"} Sep 05 06:26:38.398376 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:38.398301 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" podStartSLOduration=20.398281753 podStartE2EDuration="20.398281753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 06:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:26:21.360193338 +0000 UTC m=+5.744736469" watchObservedRunningTime="2026-09-05 06:26:38.398281753 +0000 UTC m=+22.782824878" Sep 05 06:26:38.418041 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:38.417957 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xg8xw" podStartSLOduration=3.675063748 podStartE2EDuration="22.417939562s" totalImagesPullingTime="18.742875814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 06:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:26:38.398282975 +0000 UTC m=+22.782826100" watchObservedRunningTime="2026-09-05 06:26:38.417939562 +0000 UTC m=+22.802482665" Sep 05 06:26:38.418609 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:38.418547 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dzlmk" podStartSLOduration=3.730424185 podStartE2EDuration="22.418538505s" totalImagesPullingTime="18.68811432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 06:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:26:38.418157694 +0000 UTC m=+22.802700821" watchObservedRunningTime="2026-09-05 06:26:38.418538505 +0000 UTC m=+22.803081629" Sep 05 06:26:38.439260 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:38.439206 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dtsh9" podStartSLOduration=3.769601128 podStartE2EDuration="22.439185472s" totalImagesPullingTime="18.669584344s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 06:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:26:38.438703584 +0000 UTC m=+22.823246708" watchObservedRunningTime="2026-09-05 06:26:38.439185472 +0000 UTC m=+22.823728602" Sep 05 06:26:38.478499 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:38.478449 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-m7rvs" podStartSLOduration=3.835378414 podStartE2EDuration="22.478433838s" totalImagesPullingTime="18.643055424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 06:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:26:38.478234841 +0000 UTC m=+22.862777965" watchObservedRunningTime="2026-09-05 06:26:38.478433838 +0000 UTC m=+22.862976961" Sep 05 06:26:38.497668 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:38.497618 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zqj4q" podStartSLOduration=3.870692571 podStartE2EDuration="22.497597476s" totalImagesPullingTime="18.626904905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:26:38.49748653 +0000 UTC m=+22.882029654" watchObservedRunningTime="2026-09-05 06:26:38.497597476 +0000 UTC m=+22.882140603" Sep 05 06:26:38.938958 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:38.938876 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zqj4q" Sep 05 06:26:38.939646 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:38.939617 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zqj4q" Sep 05 06:26:39.159561 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:39.159525 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:39.159723 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:39.159680 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-545p9" podUID="9103306e-ef91-4539-b914-8c3cdb23ea94" Sep 05 06:26:39.396198 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:39.395815 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" event={"ID":"000daeb0-4f4d-497e-a81e-f7457115f0e9","Type":"ContainerStarted","Data":"df2c33492c54239c6b100fb19ad58cd36d322ba99c17c2276cb04339d2d26652"} Sep 05 06:26:39.396198 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:39.395910 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" event={"ID":"000daeb0-4f4d-497e-a81e-f7457115f0e9","Type":"ContainerStarted","Data":"b4dc99b1d967842279355aee38b7e22b8505c27f87788a430a5ebc7de04132aa"} Sep 05 06:26:39.396198 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:39.395958 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" event={"ID":"000daeb0-4f4d-497e-a81e-f7457115f0e9","Type":"ContainerStarted","Data":"ba03083a1efd0045ec3a935246b589fe5898d56986a116a884d712cd24c63adb"} Sep 05 06:26:39.397045 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:39.396460 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zqj4q" Sep 05 06:26:39.397045 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:39.396710 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zqj4q" Sep 05 06:26:39.537763 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:39.537740 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 05 06:26:40.082411 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:40.082296 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-05T06:26:39.537758418Z","UUID":"2a8f741e-d66d-41b2-93fd-0086ddcb17e8","Handler":null,"Name":"","Endpoint":""} Sep 05 06:26:40.083922 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:40.083899 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 05 06:26:40.084057 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:40.083933 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 05 06:26:40.160131 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:40.159662 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:40.160131 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:40.159809 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nccpk" podUID="88f016f8-99a0-4560-b4e2-2abd92c5cf90" Sep 05 06:26:40.160131 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:40.159873 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:40.160131 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:40.160014 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzpdk" podUID="8a3094f3-d739-47e4-b8a0-49335c888b12" Sep 05 06:26:40.400604 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:40.400559 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" event={"ID":"fb143a2d-2aaa-41ee-b476-2a03fa70ea4f","Type":"ContainerStarted","Data":"a6ac68aae7efd16140e21a750a15b5542c914b644a45e470394b57afd20aa091"} Sep 05 06:26:40.401981 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:40.401923 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gf7px" event={"ID":"a1707699-6ecc-4982-9a8e-b1e22f0e751a","Type":"ContainerStarted","Data":"e745ccbfaaf04f5a969252e89b32cb92b978d07935f3fcde72db33ef9e22cd1c"} Sep 05 06:26:41.159728 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:41.159539 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:41.159882 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:41.159823 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-545p9" podUID="9103306e-ef91-4539-b914-8c3cdb23ea94" Sep 05 06:26:41.407246 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:41.407197 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" event={"ID":"fb143a2d-2aaa-41ee-b476-2a03fa70ea4f","Type":"ContainerStarted","Data":"228e096b2672532d827d27d0943d11f3743130b059432abd1610fd84be3f4663"} Sep 05 06:26:41.411263 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:41.411194 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" event={"ID":"000daeb0-4f4d-497e-a81e-f7457115f0e9","Type":"ContainerStarted","Data":"48d2413d2b49fac9f7e652c9644f8a89140db14a769ecc9b84298a6a36208b2e"} Sep 05 06:26:41.427327 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:41.427205 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gf7px" podStartSLOduration=6.811596535 podStartE2EDuration="25.427183459s" totalImagesPullingTime="18.615586924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 06:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:26:40.418967389 +0000 UTC m=+24.803510513" watchObservedRunningTime="2026-09-05 06:26:41.427183459 +0000 UTC m=+25.811726584" Sep 05 06:26:42.159740 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:42.159704 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:42.159945 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:42.159839 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nccpk" podUID="88f016f8-99a0-4560-b4e2-2abd92c5cf90" Sep 05 06:26:42.159945 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:42.159898 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:42.160069 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:42.159997 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzpdk" podUID="8a3094f3-d739-47e4-b8a0-49335c888b12" Sep 05 06:26:43.159462 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:43.159366 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:43.159737 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:43.159501 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-545p9" podUID="9103306e-ef91-4539-b914-8c3cdb23ea94" Sep 05 06:26:43.421139 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:43.420882 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" event={"ID":"000daeb0-4f4d-497e-a81e-f7457115f0e9","Type":"ContainerStarted","Data":"bcb9a0c9e19f65fccc83b75d33e01671334523862712336cd1ae473dd95b2f0c"} Sep 05 06:26:43.421291 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:43.421176 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:43.421291 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:43.421254 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:43.423381 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:43.423356 2565 generic.go:356] "Generic (PLEG): container finished" podID="5de80d59-44e0-48d2-88de-e51fdc1bd19e" containerID="659b8115097b3807f42692b1350f3616f7fb2d3cb0f54a379b213dc8f3332f84" exitCode=0 Sep 05 06:26:43.423497 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:43.423404 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ml7g5" event={"ID":"5de80d59-44e0-48d2-88de-e51fdc1bd19e","Type":"ContainerDied","Data":"659b8115097b3807f42692b1350f3616f7fb2d3cb0f54a379b213dc8f3332f84"} Sep 05 06:26:43.438595 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:43.438574 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:43.458427 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:43.458381 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" podStartSLOduration=8.813051164 podStartE2EDuration="27.458365859s" totalImagesPullingTime="18.645314695s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 06:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:26:43.457909813 +0000 UTC m=+27.842452962" watchObservedRunningTime="2026-09-05 06:26:43.458365859 +0000 UTC m=+27.842908980" Sep 05 06:26:43.458621 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:43.458601 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw864" podStartSLOduration=6.470583075 podStartE2EDuration="27.458594129s" totalImagesPullingTime="20.988011054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 06:26:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:26:41.427838534 +0000 UTC m=+25.812381658" watchObservedRunningTime="2026-09-05 06:26:43.458594129 +0000 UTC m=+27.843137250" Sep 05 06:26:44.159734 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:44.159694 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:44.160115 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:44.159737 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:44.160115 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:44.159832 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nccpk" podUID="88f016f8-99a0-4560-b4e2-2abd92c5cf90" Sep 05 06:26:44.160115 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:44.159935 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzpdk" podUID="8a3094f3-d739-47e4-b8a0-49335c888b12" Sep 05 06:26:44.428582 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:44.428547 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ml7g5" event={"ID":"5de80d59-44e0-48d2-88de-e51fdc1bd19e","Type":"ContainerStarted","Data":"480577c0cc3c814ef7f6e4830ea9d61a003e8d47deb8e812a40cd952b5e9661b"} Sep 05 06:26:44.429242 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:44.429172 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:44.447985 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:44.447959 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:26:44.984273 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:44.984232 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qzpdk"] Sep 05 06:26:44.984456 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:44.984373 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:44.984497 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:44.984466 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzpdk" podUID="8a3094f3-d739-47e4-b8a0-49335c888b12" Sep 05 06:26:44.987138 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:44.987114 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nccpk"] Sep 05 06:26:44.987343 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:44.987215 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:44.987343 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:44.987304 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nccpk" podUID="88f016f8-99a0-4560-b4e2-2abd92c5cf90" Sep 05 06:26:44.987815 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:44.987786 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-545p9"] Sep 05 06:26:44.987915 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:44.987902 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:44.988025 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:44.988009 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-545p9" podUID="9103306e-ef91-4539-b914-8c3cdb23ea94" Sep 05 06:26:45.433002 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:45.432965 2565 generic.go:356] "Generic (PLEG): container finished" podID="5de80d59-44e0-48d2-88de-e51fdc1bd19e" containerID="480577c0cc3c814ef7f6e4830ea9d61a003e8d47deb8e812a40cd952b5e9661b" exitCode=0 Sep 05 06:26:45.433757 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:45.433014 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ml7g5" event={"ID":"5de80d59-44e0-48d2-88de-e51fdc1bd19e","Type":"ContainerDied","Data":"480577c0cc3c814ef7f6e4830ea9d61a003e8d47deb8e812a40cd952b5e9661b"} Sep 05 06:26:46.160469 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:46.160433 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:46.160640 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:46.160558 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-545p9" podUID="9103306e-ef91-4539-b914-8c3cdb23ea94" Sep 05 06:26:47.159888 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:47.159698 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:47.160204 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:47.159698 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:47.160204 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:47.159975 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzpdk" podUID="8a3094f3-d739-47e4-b8a0-49335c888b12" Sep 05 06:26:47.160204 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:47.160033 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nccpk" podUID="88f016f8-99a0-4560-b4e2-2abd92c5cf90" Sep 05 06:26:47.440453 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:47.440419 2565 generic.go:356] "Generic (PLEG): container finished" podID="5de80d59-44e0-48d2-88de-e51fdc1bd19e" containerID="9d492df2618cad4d15f367a778047513fc263ea8dfba9106c62e33d3e8d9dabd" exitCode=0 Sep 05 06:26:47.440607 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:47.440458 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ml7g5" event={"ID":"5de80d59-44e0-48d2-88de-e51fdc1bd19e","Type":"ContainerDied","Data":"9d492df2618cad4d15f367a778047513fc263ea8dfba9106c62e33d3e8d9dabd"} Sep 05 06:26:48.159600 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:48.159565 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:48.159750 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:48.159680 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-545p9" podUID="9103306e-ef91-4539-b914-8c3cdb23ea94" Sep 05 06:26:49.160083 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:49.160045 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:49.160538 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:49.160045 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:49.160538 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:49.160166 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzpdk" podUID="8a3094f3-d739-47e4-b8a0-49335c888b12" Sep 05 06:26:49.160538 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:49.160281 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nccpk" podUID="88f016f8-99a0-4560-b4e2-2abd92c5cf90" Sep 05 06:26:49.882163 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:49.882126 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3094f3-d739-47e4-b8a0-49335c888b12-original-pull-secret\") pod \"global-pull-secret-syncer-qzpdk\" (UID: \"8a3094f3-d739-47e4-b8a0-49335c888b12\") " pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:49.882385 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:49.882192 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs\") pod \"network-metrics-daemon-545p9\" (UID: \"9103306e-ef91-4539-b914-8c3cdb23ea94\") " pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:49.882385 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:49.882281 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 05 06:26:49.882385 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:49.882297 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 06:26:49.882385 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:49.882359 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a3094f3-d739-47e4-b8a0-49335c888b12-original-pull-secret podName:8a3094f3-d739-47e4-b8a0-49335c888b12 nodeName:}" failed. No retries permitted until 2026-09-05 06:27:21.882343841 +0000 UTC m=+66.266886942 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a3094f3-d739-47e4-b8a0-49335c888b12-original-pull-secret") pod "global-pull-secret-syncer-qzpdk" (UID: "8a3094f3-d739-47e4-b8a0-49335c888b12") : object "kube-system"/"original-pull-secret" not registered Sep 05 06:26:49.882385 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:49.882380 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs podName:9103306e-ef91-4539-b914-8c3cdb23ea94 nodeName:}" failed. No retries permitted until 2026-09-05 06:27:21.882369963 +0000 UTC m=+66.266913070 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs") pod "network-metrics-daemon-545p9" (UID: "9103306e-ef91-4539-b914-8c3cdb23ea94") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 06:26:49.983553 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:49.983515 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ch4c\" (UniqueName: \"kubernetes.io/projected/88f016f8-99a0-4560-b4e2-2abd92c5cf90-kube-api-access-8ch4c\") pod \"network-check-target-nccpk\" (UID: \"88f016f8-99a0-4560-b4e2-2abd92c5cf90\") " pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:49.983741 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:49.983671 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 05 06:26:49.983741 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:49.983700 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 05 06:26:49.983741 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:49.983714 2565 projected.go:196] Error preparing data for projected volume kube-api-access-8ch4c for pod openshift-network-diagnostics/network-check-target-nccpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 06:26:49.983852 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:49.983779 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88f016f8-99a0-4560-b4e2-2abd92c5cf90-kube-api-access-8ch4c podName:88f016f8-99a0-4560-b4e2-2abd92c5cf90 nodeName:}" failed. No retries permitted until 2026-09-05 06:27:21.983759334 +0000 UTC m=+66.368302448 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8ch4c" (UniqueName: "kubernetes.io/projected/88f016f8-99a0-4560-b4e2-2abd92c5cf90-kube-api-access-8ch4c") pod "network-check-target-nccpk" (UID: "88f016f8-99a0-4560-b4e2-2abd92c5cf90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 06:26:50.162857 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:50.162781 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:50.163276 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:50.162916 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-545p9" podUID="9103306e-ef91-4539-b914-8c3cdb23ea94" Sep 05 06:26:50.893346 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:50.893245 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeReady" Sep 05 06:26:50.893512 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:50.893413 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 05 06:26:50.967379 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:50.967346 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-j4dq5"] Sep 05 06:26:50.970209 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:50.970177 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cmxtg"] Sep 05 06:26:50.970366 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:50.970342 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j4dq5" Sep 05 06:26:50.974411 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:50.974386 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v76tl\"" Sep 05 06:26:50.974679 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:50.974386 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 05 06:26:50.974791 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:50.974392 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 05 06:26:50.975683 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:50.975663 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cmxtg" Sep 05 06:26:50.978096 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:50.978075 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 05 06:26:50.978394 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:50.978376 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qnm9m\"" Sep 05 06:26:50.978476 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:50.978402 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 05 06:26:50.978656 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:50.978643 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 05 06:26:51.007264 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.007214 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cmxtg"] Sep 05 06:26:51.007511 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.007285 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j4dq5"] Sep 05 06:26:51.091858 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.091822 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w5cd\" (UniqueName: \"kubernetes.io/projected/3f8f4891-a720-43c1-bdf3-73b8250fde4f-kube-api-access-8w5cd\") pod \"dns-default-j4dq5\" (UID: \"3f8f4891-a720-43c1-bdf3-73b8250fde4f\") " pod="openshift-dns/dns-default-j4dq5" Sep 05 06:26:51.092051 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.091883 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8f4891-a720-43c1-bdf3-73b8250fde4f-config-volume\") pod \"dns-default-j4dq5\" (UID: \"3f8f4891-a720-43c1-bdf3-73b8250fde4f\") " pod="openshift-dns/dns-default-j4dq5" Sep 05 06:26:51.092051 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.091963 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ffad8c9d-5409-4c11-a250-686bb69099d4-cert\") pod \"ingress-canary-cmxtg\" (UID: \"ffad8c9d-5409-4c11-a250-686bb69099d4\") " pod="openshift-ingress-canary/ingress-canary-cmxtg" Sep 05 06:26:51.092051 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.091994 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f8f4891-a720-43c1-bdf3-73b8250fde4f-metrics-tls\") pod \"dns-default-j4dq5\" (UID: \"3f8f4891-a720-43c1-bdf3-73b8250fde4f\") " pod="openshift-dns/dns-default-j4dq5" Sep 05 06:26:51.092051 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.092022 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f8f4891-a720-43c1-bdf3-73b8250fde4f-tmp-dir\") pod \"dns-default-j4dq5\" (UID: \"3f8f4891-a720-43c1-bdf3-73b8250fde4f\") " pod="openshift-dns/dns-default-j4dq5" Sep 05 06:26:51.092271 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.092058 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpccd\" (UniqueName: \"kubernetes.io/projected/ffad8c9d-5409-4c11-a250-686bb69099d4-kube-api-access-wpccd\") pod \"ingress-canary-cmxtg\" (UID: \"ffad8c9d-5409-4c11-a250-686bb69099d4\") " pod="openshift-ingress-canary/ingress-canary-cmxtg" Sep 05 06:26:51.160176 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.160080 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:26:51.160445 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.160416 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:26:51.164151 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.163862 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 05 06:26:51.164151 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.163976 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-24sqj\"" Sep 05 06:26:51.164151 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.163995 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 05 06:26:51.164151 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.164106 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 05 06:26:51.192658 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.192622 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpccd\" (UniqueName: \"kubernetes.io/projected/ffad8c9d-5409-4c11-a250-686bb69099d4-kube-api-access-wpccd\") pod \"ingress-canary-cmxtg\" (UID: \"ffad8c9d-5409-4c11-a250-686bb69099d4\") " pod="openshift-ingress-canary/ingress-canary-cmxtg" Sep 05 06:26:51.192864 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.192690 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w5cd\" (UniqueName: \"kubernetes.io/projected/3f8f4891-a720-43c1-bdf3-73b8250fde4f-kube-api-access-8w5cd\") pod \"dns-default-j4dq5\" (UID: \"3f8f4891-a720-43c1-bdf3-73b8250fde4f\") " pod="openshift-dns/dns-default-j4dq5" Sep 05 06:26:51.192864 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.192730 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8f4891-a720-43c1-bdf3-73b8250fde4f-config-volume\") pod \"dns-default-j4dq5\" (UID: \"3f8f4891-a720-43c1-bdf3-73b8250fde4f\") " pod="openshift-dns/dns-default-j4dq5" Sep 05 06:26:51.192864 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.192779 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ffad8c9d-5409-4c11-a250-686bb69099d4-cert\") pod \"ingress-canary-cmxtg\" (UID: \"ffad8c9d-5409-4c11-a250-686bb69099d4\") " pod="openshift-ingress-canary/ingress-canary-cmxtg" Sep 05 06:26:51.192864 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.192803 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f8f4891-a720-43c1-bdf3-73b8250fde4f-metrics-tls\") pod \"dns-default-j4dq5\" (UID: \"3f8f4891-a720-43c1-bdf3-73b8250fde4f\") " pod="openshift-dns/dns-default-j4dq5" Sep 05 06:26:51.192864 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.192824 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f8f4891-a720-43c1-bdf3-73b8250fde4f-tmp-dir\") pod \"dns-default-j4dq5\" (UID: \"3f8f4891-a720-43c1-bdf3-73b8250fde4f\") " pod="openshift-dns/dns-default-j4dq5" Sep 05 06:26:51.193178 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.193158 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f8f4891-a720-43c1-bdf3-73b8250fde4f-tmp-dir\") pod \"dns-default-j4dq5\" (UID: \"3f8f4891-a720-43c1-bdf3-73b8250fde4f\") " pod="openshift-dns/dns-default-j4dq5" Sep 05 06:26:51.193245 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:51.193205 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 05 06:26:51.193297 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:51.193273 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f8f4891-a720-43c1-bdf3-73b8250fde4f-metrics-tls podName:3f8f4891-a720-43c1-bdf3-73b8250fde4f nodeName:}" failed. No retries permitted until 2026-09-05 06:26:51.693251596 +0000 UTC m=+36.077794712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f8f4891-a720-43c1-bdf3-73b8250fde4f-metrics-tls") pod "dns-default-j4dq5" (UID: "3f8f4891-a720-43c1-bdf3-73b8250fde4f") : secret "dns-default-metrics-tls" not found Sep 05 06:26:51.193657 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.193625 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8f4891-a720-43c1-bdf3-73b8250fde4f-config-volume\") pod \"dns-default-j4dq5\" (UID: \"3f8f4891-a720-43c1-bdf3-73b8250fde4f\") " pod="openshift-dns/dns-default-j4dq5" Sep 05 06:26:51.197936 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.197915 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ffad8c9d-5409-4c11-a250-686bb69099d4-cert\") pod \"ingress-canary-cmxtg\" (UID: \"ffad8c9d-5409-4c11-a250-686bb69099d4\") " pod="openshift-ingress-canary/ingress-canary-cmxtg" Sep 05 06:26:51.205399 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.205367 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w5cd\" (UniqueName: \"kubernetes.io/projected/3f8f4891-a720-43c1-bdf3-73b8250fde4f-kube-api-access-8w5cd\") pod \"dns-default-j4dq5\" (UID: \"3f8f4891-a720-43c1-bdf3-73b8250fde4f\") " pod="openshift-dns/dns-default-j4dq5" Sep 05 06:26:51.206110 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.206087 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpccd\" (UniqueName: \"kubernetes.io/projected/ffad8c9d-5409-4c11-a250-686bb69099d4-kube-api-access-wpccd\") pod \"ingress-canary-cmxtg\" (UID: \"ffad8c9d-5409-4c11-a250-686bb69099d4\") " pod="openshift-ingress-canary/ingress-canary-cmxtg" Sep 05 06:26:51.291871 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.291834 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cmxtg" Sep 05 06:26:51.696481 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:51.696446 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f8f4891-a720-43c1-bdf3-73b8250fde4f-metrics-tls\") pod \"dns-default-j4dq5\" (UID: \"3f8f4891-a720-43c1-bdf3-73b8250fde4f\") " pod="openshift-dns/dns-default-j4dq5" Sep 05 06:26:51.696682 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:51.696622 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 05 06:26:51.696745 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:51.696704 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f8f4891-a720-43c1-bdf3-73b8250fde4f-metrics-tls podName:3f8f4891-a720-43c1-bdf3-73b8250fde4f nodeName:}" failed. No retries permitted until 2026-09-05 06:26:52.696681407 +0000 UTC m=+37.081224533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f8f4891-a720-43c1-bdf3-73b8250fde4f-metrics-tls") pod "dns-default-j4dq5" (UID: "3f8f4891-a720-43c1-bdf3-73b8250fde4f") : secret "dns-default-metrics-tls" not found Sep 05 06:26:52.113511 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:52.113341 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cmxtg"] Sep 05 06:26:52.126472 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:26:52.126339 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffad8c9d_5409_4c11_a250_686bb69099d4.slice/crio-4be4a5c0cec039548f5ea75e32fb7e9da28f88857568c9d9ef4d02172856bf7d WatchSource:0}: Error finding container 4be4a5c0cec039548f5ea75e32fb7e9da28f88857568c9d9ef4d02172856bf7d: Status 404 returned error can't find the container with id 4be4a5c0cec039548f5ea75e32fb7e9da28f88857568c9d9ef4d02172856bf7d Sep 05 06:26:52.160035 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:52.160009 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:26:52.162803 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:52.162782 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 05 06:26:52.162925 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:52.162814 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4m5kh\"" Sep 05 06:26:52.456041 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:52.456009 2565 generic.go:356] "Generic (PLEG): container finished" podID="5de80d59-44e0-48d2-88de-e51fdc1bd19e" containerID="cd5f1c0495e52b38b23515539af82ad557f08d192b9b4dd170a6adc3ca20a70a" exitCode=0 Sep 05 06:26:52.456618 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:52.456086 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ml7g5" event={"ID":"5de80d59-44e0-48d2-88de-e51fdc1bd19e","Type":"ContainerDied","Data":"cd5f1c0495e52b38b23515539af82ad557f08d192b9b4dd170a6adc3ca20a70a"} Sep 05 06:26:52.457334 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:52.457286 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cmxtg" event={"ID":"ffad8c9d-5409-4c11-a250-686bb69099d4","Type":"ContainerStarted","Data":"4be4a5c0cec039548f5ea75e32fb7e9da28f88857568c9d9ef4d02172856bf7d"} Sep 05 06:26:52.703876 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:52.703831 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f8f4891-a720-43c1-bdf3-73b8250fde4f-metrics-tls\") pod \"dns-default-j4dq5\" (UID: \"3f8f4891-a720-43c1-bdf3-73b8250fde4f\") " pod="openshift-dns/dns-default-j4dq5" Sep 05 06:26:52.704063 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:52.704028 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 05 06:26:52.704125 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:52.704102 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f8f4891-a720-43c1-bdf3-73b8250fde4f-metrics-tls podName:3f8f4891-a720-43c1-bdf3-73b8250fde4f nodeName:}" failed. No retries permitted until 2026-09-05 06:26:54.704080164 +0000 UTC m=+39.088623286 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f8f4891-a720-43c1-bdf3-73b8250fde4f-metrics-tls") pod "dns-default-j4dq5" (UID: "3f8f4891-a720-43c1-bdf3-73b8250fde4f") : secret "dns-default-metrics-tls" not found Sep 05 06:26:53.462512 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:53.462475 2565 generic.go:356] "Generic (PLEG): container finished" podID="5de80d59-44e0-48d2-88de-e51fdc1bd19e" containerID="18fc76d8dc99508cc6aee987320332f4fa60c5632657214899c4c760067961ee" exitCode=0 Sep 05 06:26:53.463081 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:53.462521 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ml7g5" event={"ID":"5de80d59-44e0-48d2-88de-e51fdc1bd19e","Type":"ContainerDied","Data":"18fc76d8dc99508cc6aee987320332f4fa60c5632657214899c4c760067961ee"} Sep 05 06:26:54.468291 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:54.468255 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ml7g5" event={"ID":"5de80d59-44e0-48d2-88de-e51fdc1bd19e","Type":"ContainerStarted","Data":"b66ef6e17ef8cfc4f66697967587166053e8e519c50c4c0b67af2c5ce442dc6c"} Sep 05 06:26:54.495704 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:54.495657 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ml7g5" podStartSLOduration=8.723009527 podStartE2EDuration="38.495641327s" totalImagesPullingTime="29.7726318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 06:26:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:26:54.494466877 +0000 UTC m=+38.879010000" watchObservedRunningTime="2026-09-05 06:26:54.495641327 +0000 UTC m=+38.880184451" Sep 05 06:26:54.720597 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:54.720504 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f8f4891-a720-43c1-bdf3-73b8250fde4f-metrics-tls\") pod \"dns-default-j4dq5\" (UID: \"3f8f4891-a720-43c1-bdf3-73b8250fde4f\") " pod="openshift-dns/dns-default-j4dq5" Sep 05 06:26:54.720748 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:54.720658 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 05 06:26:54.720748 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:54.720724 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f8f4891-a720-43c1-bdf3-73b8250fde4f-metrics-tls podName:3f8f4891-a720-43c1-bdf3-73b8250fde4f nodeName:}" failed. No retries permitted until 2026-09-05 06:26:58.720707 +0000 UTC m=+43.105250130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f8f4891-a720-43c1-bdf3-73b8250fde4f-metrics-tls") pod "dns-default-j4dq5" (UID: "3f8f4891-a720-43c1-bdf3-73b8250fde4f") : secret "dns-default-metrics-tls" not found Sep 05 06:26:55.472544 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:55.472505 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cmxtg" event={"ID":"ffad8c9d-5409-4c11-a250-686bb69099d4","Type":"ContainerStarted","Data":"c4429207647b998fc9d64a42f0fb9d27a0cd4f009ab92d83d107d0465502e55a"} Sep 05 06:26:55.490530 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:55.490474 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cmxtg" podStartSLOduration=3.078902862 podStartE2EDuration="5.490456785s" totalImagesPullingTime="2.411553923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:26:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:26:55.489924594 +0000 UTC m=+39.874467717" watchObservedRunningTime="2026-09-05 06:26:55.490456785 +0000 UTC m=+39.874999906" Sep 05 06:26:58.748134 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:26:58.748089 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f8f4891-a720-43c1-bdf3-73b8250fde4f-metrics-tls\") pod \"dns-default-j4dq5\" (UID: \"3f8f4891-a720-43c1-bdf3-73b8250fde4f\") " pod="openshift-dns/dns-default-j4dq5" Sep 05 06:26:58.748550 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:58.748269 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 05 06:26:58.748550 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:26:58.748360 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f8f4891-a720-43c1-bdf3-73b8250fde4f-metrics-tls podName:3f8f4891-a720-43c1-bdf3-73b8250fde4f nodeName:}" failed. No retries permitted until 2026-09-05 06:27:06.748343517 +0000 UTC m=+51.132886636 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f8f4891-a720-43c1-bdf3-73b8250fde4f-metrics-tls") pod "dns-default-j4dq5" (UID: "3f8f4891-a720-43c1-bdf3-73b8250fde4f") : secret "dns-default-metrics-tls" not found Sep 05 06:27:06.805291 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:06.805248 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f8f4891-a720-43c1-bdf3-73b8250fde4f-metrics-tls\") pod \"dns-default-j4dq5\" (UID: \"3f8f4891-a720-43c1-bdf3-73b8250fde4f\") " pod="openshift-dns/dns-default-j4dq5" Sep 05 06:27:06.805750 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:06.805393 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 05 06:27:06.805750 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:06.805458 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f8f4891-a720-43c1-bdf3-73b8250fde4f-metrics-tls podName:3f8f4891-a720-43c1-bdf3-73b8250fde4f nodeName:}" failed. No retries permitted until 2026-09-05 06:27:22.805443367 +0000 UTC m=+67.189986482 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f8f4891-a720-43c1-bdf3-73b8250fde4f-metrics-tls") pod "dns-default-j4dq5" (UID: "3f8f4891-a720-43c1-bdf3-73b8250fde4f") : secret "dns-default-metrics-tls" not found Sep 05 06:27:16.453022 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:16.452991 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6mvpl" Sep 05 06:27:21.909162 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:21.909108 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3094f3-d739-47e4-b8a0-49335c888b12-original-pull-secret\") pod \"global-pull-secret-syncer-qzpdk\" (UID: \"8a3094f3-d739-47e4-b8a0-49335c888b12\") " pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:27:21.909624 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:21.909184 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs\") pod \"network-metrics-daemon-545p9\" (UID: \"9103306e-ef91-4539-b914-8c3cdb23ea94\") " pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:27:21.911936 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:21.911912 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 05 06:27:21.912013 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:21.911956 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 05 06:27:21.919950 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:21.919931 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 05 06:27:21.920011 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:21.919989 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs podName:9103306e-ef91-4539-b914-8c3cdb23ea94 nodeName:}" failed. No retries permitted until 2026-09-05 06:28:25.919972403 +0000 UTC m=+130.304515506 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs") pod "network-metrics-daemon-545p9" (UID: "9103306e-ef91-4539-b914-8c3cdb23ea94") : secret "metrics-daemon-secret" not found Sep 05 06:27:21.922961 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:21.922940 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3094f3-d739-47e4-b8a0-49335c888b12-original-pull-secret\") pod \"global-pull-secret-syncer-qzpdk\" (UID: \"8a3094f3-d739-47e4-b8a0-49335c888b12\") " pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:27:22.010569 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:22.010515 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ch4c\" (UniqueName: \"kubernetes.io/projected/88f016f8-99a0-4560-b4e2-2abd92c5cf90-kube-api-access-8ch4c\") pod \"network-check-target-nccpk\" (UID: \"88f016f8-99a0-4560-b4e2-2abd92c5cf90\") " pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:27:22.013570 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:22.013552 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 05 06:27:22.024272 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:22.024247 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 05 06:27:22.034830 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:22.034803 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ch4c\" (UniqueName: \"kubernetes.io/projected/88f016f8-99a0-4560-b4e2-2abd92c5cf90-kube-api-access-8ch4c\") pod \"network-check-target-nccpk\" (UID: \"88f016f8-99a0-4560-b4e2-2abd92c5cf90\") " pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:27:22.073792 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:22.073753 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzpdk" Sep 05 06:27:22.082497 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:22.082474 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-24sqj\"" Sep 05 06:27:22.090769 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:22.090749 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:27:22.346119 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:22.346037 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qzpdk"] Sep 05 06:27:22.349326 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:27:22.349284 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3094f3_d739_47e4_b8a0_49335c888b12.slice/crio-8270de1f3ec01ea16656ecac59b8e91366e2d7fe153ea349e04795fcedc34815 WatchSource:0}: Error finding container 8270de1f3ec01ea16656ecac59b8e91366e2d7fe153ea349e04795fcedc34815: Status 404 returned error can't find the container with id 8270de1f3ec01ea16656ecac59b8e91366e2d7fe153ea349e04795fcedc34815 Sep 05 06:27:22.366091 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:22.366069 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nccpk"] Sep 05 06:27:22.369184 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:27:22.369160 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f016f8_99a0_4560_b4e2_2abd92c5cf90.slice/crio-4ae5c27268fc21c0a4897f28b2dfd74f5df1e97ccb7dbbc8238aea5e9796e88d WatchSource:0}: Error finding container 4ae5c27268fc21c0a4897f28b2dfd74f5df1e97ccb7dbbc8238aea5e9796e88d: Status 404 returned error can't find the container with id 4ae5c27268fc21c0a4897f28b2dfd74f5df1e97ccb7dbbc8238aea5e9796e88d Sep 05 06:27:22.532882 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:22.532846 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nccpk" event={"ID":"88f016f8-99a0-4560-b4e2-2abd92c5cf90","Type":"ContainerStarted","Data":"4ae5c27268fc21c0a4897f28b2dfd74f5df1e97ccb7dbbc8238aea5e9796e88d"} Sep 05 06:27:22.533975 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:22.533951 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qzpdk" event={"ID":"8a3094f3-d739-47e4-b8a0-49335c888b12","Type":"ContainerStarted","Data":"8270de1f3ec01ea16656ecac59b8e91366e2d7fe153ea349e04795fcedc34815"} Sep 05 06:27:22.817976 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:22.817937 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f8f4891-a720-43c1-bdf3-73b8250fde4f-metrics-tls\") pod \"dns-default-j4dq5\" (UID: \"3f8f4891-a720-43c1-bdf3-73b8250fde4f\") " pod="openshift-dns/dns-default-j4dq5" Sep 05 06:27:22.818148 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:22.818067 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 05 06:27:22.818148 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:22.818122 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f8f4891-a720-43c1-bdf3-73b8250fde4f-metrics-tls podName:3f8f4891-a720-43c1-bdf3-73b8250fde4f nodeName:}" failed. No retries permitted until 2026-09-05 06:27:54.818107563 +0000 UTC m=+99.202650678 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f8f4891-a720-43c1-bdf3-73b8250fde4f-metrics-tls") pod "dns-default-j4dq5" (UID: "3f8f4891-a720-43c1-bdf3-73b8250fde4f") : secret "dns-default-metrics-tls" not found Sep 05 06:27:27.549904 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:27.549858 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nccpk" event={"ID":"88f016f8-99a0-4560-b4e2-2abd92c5cf90","Type":"ContainerStarted","Data":"0c300d6e1d03c3c6199b3e502b41e97b1268ccaac3d30047b072a41db0c18a15"} Sep 05 06:27:27.550430 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:27.549921 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:27:27.551290 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:27.551266 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qzpdk" event={"ID":"8a3094f3-d739-47e4-b8a0-49335c888b12","Type":"ContainerStarted","Data":"44f27a4b9f4be8b0345343ebbbacb5c5c45b0b3178d19733c7fe90eaa3653b57"} Sep 05 06:27:27.565655 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:27.565602 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nccpk" podStartSLOduration=66.930306386 podStartE2EDuration="1m11.565586236s" totalImagesPullingTime="4.63527985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:27:27.56485338 +0000 UTC m=+71.949396504" watchObservedRunningTime="2026-09-05 06:27:27.565586236 +0000 UTC m=+71.950129360" Sep 05 06:27:27.580541 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:27.580487 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qzpdk" podStartSLOduration=66.921868177 podStartE2EDuration="1m11.580471233s" totalImagesPullingTime="4.658603056s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 06:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:27:27.579638033 +0000 UTC m=+71.964181169" watchObservedRunningTime="2026-09-05 06:27:27.580471233 +0000 UTC m=+71.965014356" Sep 05 06:27:28.031178 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.031144 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k68n5"] Sep 05 06:27:28.033093 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.033078 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k68n5" Sep 05 06:27:28.035494 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.035470 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 05 06:27:28.035613 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.035468 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 05 06:27:28.036283 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.036251 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 05 06:27:28.036414 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.036287 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-85vbl\"" Sep 05 06:27:28.036414 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.036298 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 05 06:27:28.043482 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.043463 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k68n5"] Sep 05 06:27:28.059011 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.058984 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62636c45-1709-4102-bbfc-a13e5f699d1e-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-k68n5\" (UID: \"62636c45-1709-4102-bbfc-a13e5f699d1e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k68n5" Sep 05 06:27:28.059147 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.059019 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmmdp\" (UniqueName: \"kubernetes.io/projected/62636c45-1709-4102-bbfc-a13e5f699d1e-kube-api-access-vmmdp\") pod \"kube-storage-version-migrator-operator-8d7dd4675-k68n5\" (UID: \"62636c45-1709-4102-bbfc-a13e5f699d1e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k68n5" Sep 05 06:27:28.059147 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.059040 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62636c45-1709-4102-bbfc-a13e5f699d1e-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-k68n5\" (UID: \"62636c45-1709-4102-bbfc-a13e5f699d1e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k68n5" Sep 05 06:27:28.150563 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.150528 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nx5pm"] Sep 05 06:27:28.152500 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.152485 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nx5pm" Sep 05 06:27:28.155211 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.155172 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 05 06:27:28.155387 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.155217 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 05 06:27:28.155387 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.155222 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 05 06:27:28.155387 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.155230 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-ckjcq\"" Sep 05 06:27:28.159334 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.159299 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62636c45-1709-4102-bbfc-a13e5f699d1e-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-k68n5\" (UID: \"62636c45-1709-4102-bbfc-a13e5f699d1e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k68n5" Sep 05 06:27:28.159451 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.159357 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmmdp\" (UniqueName: \"kubernetes.io/projected/62636c45-1709-4102-bbfc-a13e5f699d1e-kube-api-access-vmmdp\") pod \"kube-storage-version-migrator-operator-8d7dd4675-k68n5\" (UID: \"62636c45-1709-4102-bbfc-a13e5f699d1e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k68n5" Sep 05 06:27:28.159451 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.159389 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0a59b50-b545-4ea3-ac55-5906910b22c7-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nx5pm\" (UID: \"a0a59b50-b545-4ea3-ac55-5906910b22c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nx5pm" Sep 05 06:27:28.159451 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.159417 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62636c45-1709-4102-bbfc-a13e5f699d1e-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-k68n5\" (UID: \"62636c45-1709-4102-bbfc-a13e5f699d1e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k68n5" Sep 05 06:27:28.159580 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.159556 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6v8s\" (UniqueName: \"kubernetes.io/projected/a0a59b50-b545-4ea3-ac55-5906910b22c7-kube-api-access-h6v8s\") pod \"cluster-samples-operator-7cdcfcf76c-nx5pm\" (UID: \"a0a59b50-b545-4ea3-ac55-5906910b22c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nx5pm" Sep 05 06:27:28.160061 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.160034 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62636c45-1709-4102-bbfc-a13e5f699d1e-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-k68n5\" (UID: \"62636c45-1709-4102-bbfc-a13e5f699d1e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k68n5" Sep 05 06:27:28.162097 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.162080 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62636c45-1709-4102-bbfc-a13e5f699d1e-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-k68n5\" (UID: \"62636c45-1709-4102-bbfc-a13e5f699d1e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k68n5" Sep 05 06:27:28.163211 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.163194 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nx5pm"] Sep 05 06:27:28.168131 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.168104 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmmdp\" (UniqueName: \"kubernetes.io/projected/62636c45-1709-4102-bbfc-a13e5f699d1e-kube-api-access-vmmdp\") pod \"kube-storage-version-migrator-operator-8d7dd4675-k68n5\" (UID: \"62636c45-1709-4102-bbfc-a13e5f699d1e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k68n5" Sep 05 06:27:28.260073 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.260036 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6v8s\" (UniqueName: \"kubernetes.io/projected/a0a59b50-b545-4ea3-ac55-5906910b22c7-kube-api-access-h6v8s\") pod \"cluster-samples-operator-7cdcfcf76c-nx5pm\" (UID: \"a0a59b50-b545-4ea3-ac55-5906910b22c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nx5pm" Sep 05 06:27:28.260073 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.260077 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0a59b50-b545-4ea3-ac55-5906910b22c7-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nx5pm\" (UID: \"a0a59b50-b545-4ea3-ac55-5906910b22c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nx5pm" Sep 05 06:27:28.260440 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:28.260175 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 05 06:27:28.260440 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:28.260234 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0a59b50-b545-4ea3-ac55-5906910b22c7-samples-operator-tls podName:a0a59b50-b545-4ea3-ac55-5906910b22c7 nodeName:}" failed. No retries permitted until 2026-09-05 06:27:28.760218211 +0000 UTC m=+73.144761313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a0a59b50-b545-4ea3-ac55-5906910b22c7-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-nx5pm" (UID: "a0a59b50-b545-4ea3-ac55-5906910b22c7") : secret "samples-operator-tls" not found Sep 05 06:27:28.271805 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.271776 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6v8s\" (UniqueName: \"kubernetes.io/projected/a0a59b50-b545-4ea3-ac55-5906910b22c7-kube-api-access-h6v8s\") pod \"cluster-samples-operator-7cdcfcf76c-nx5pm\" (UID: \"a0a59b50-b545-4ea3-ac55-5906910b22c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nx5pm" Sep 05 06:27:28.341745 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.341654 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k68n5" Sep 05 06:27:28.356970 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.356943 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-q8tzt"] Sep 05 06:27:28.358832 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.358812 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-q8tzt" Sep 05 06:27:28.360745 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.360721 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01815101-b04b-43ea-bb6c-bc896fd4b969-config\") pod \"service-ca-operator-859f94c4b-q8tzt\" (UID: \"01815101-b04b-43ea-bb6c-bc896fd4b969\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-q8tzt" Sep 05 06:27:28.360869 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.360779 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01815101-b04b-43ea-bb6c-bc896fd4b969-serving-cert\") pod \"service-ca-operator-859f94c4b-q8tzt\" (UID: \"01815101-b04b-43ea-bb6c-bc896fd4b969\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-q8tzt" Sep 05 06:27:28.360869 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.360821 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvf54\" (UniqueName: \"kubernetes.io/projected/01815101-b04b-43ea-bb6c-bc896fd4b969-kube-api-access-zvf54\") pod \"service-ca-operator-859f94c4b-q8tzt\" (UID: \"01815101-b04b-43ea-bb6c-bc896fd4b969\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-q8tzt" Sep 05 06:27:28.361700 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.361677 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7bd7fcc894-s297m"] Sep 05 06:27:28.362218 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.362200 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 05 06:27:28.362330 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.362247 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-6lc45\"" Sep 05 06:27:28.362544 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.362528 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 05 06:27:28.362624 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.362607 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 05 06:27:28.362686 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.362621 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 05 06:27:28.363987 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.363969 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:28.366564 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.366497 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 05 06:27:28.366676 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.366626 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rfhpd\"" Sep 05 06:27:28.366935 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.366921 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 05 06:27:28.367179 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.367159 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 05 06:27:28.371916 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.371896 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 05 06:27:28.393826 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.393691 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-q8tzt"] Sep 05 06:27:28.397865 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.397834 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bd7fcc894-s297m"] Sep 05 06:27:28.461781 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.461746 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01815101-b04b-43ea-bb6c-bc896fd4b969-config\") pod \"service-ca-operator-859f94c4b-q8tzt\" (UID: \"01815101-b04b-43ea-bb6c-bc896fd4b969\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-q8tzt" Sep 05 06:27:28.461781 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.461787 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6653acf2-739a-4628-8f9f-e0658b4a9338-trusted-ca\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:28.462004 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.461819 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01815101-b04b-43ea-bb6c-bc896fd4b969-serving-cert\") pod \"service-ca-operator-859f94c4b-q8tzt\" (UID: \"01815101-b04b-43ea-bb6c-bc896fd4b969\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-q8tzt" Sep 05 06:27:28.462004 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.461844 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvf54\" (UniqueName: \"kubernetes.io/projected/01815101-b04b-43ea-bb6c-bc896fd4b969-kube-api-access-zvf54\") pod \"service-ca-operator-859f94c4b-q8tzt\" (UID: \"01815101-b04b-43ea-bb6c-bc896fd4b969\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-q8tzt" Sep 05 06:27:28.462004 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.461863 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-certificates\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:28.462004 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.461887 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-tls\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:28.462004 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.461908 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6653acf2-739a-4628-8f9f-e0658b4a9338-ca-trust-extracted\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:28.462004 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.461929 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6653acf2-739a-4628-8f9f-e0658b4a9338-installation-pull-secrets\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:28.462004 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.461953 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-bound-sa-token\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:28.462258 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.462129 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6653acf2-739a-4628-8f9f-e0658b4a9338-image-registry-private-configuration\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:28.462258 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.462174 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f99bb\" (UniqueName: \"kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-kube-api-access-f99bb\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:28.462390 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.462371 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01815101-b04b-43ea-bb6c-bc896fd4b969-config\") pod \"service-ca-operator-859f94c4b-q8tzt\" (UID: \"01815101-b04b-43ea-bb6c-bc896fd4b969\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-q8tzt" Sep 05 06:27:28.464140 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.464122 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01815101-b04b-43ea-bb6c-bc896fd4b969-serving-cert\") pod \"service-ca-operator-859f94c4b-q8tzt\" (UID: \"01815101-b04b-43ea-bb6c-bc896fd4b969\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-q8tzt" Sep 05 06:27:28.470281 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.470252 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvf54\" (UniqueName: \"kubernetes.io/projected/01815101-b04b-43ea-bb6c-bc896fd4b969-kube-api-access-zvf54\") pod \"service-ca-operator-859f94c4b-q8tzt\" (UID: \"01815101-b04b-43ea-bb6c-bc896fd4b969\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-q8tzt" Sep 05 06:27:28.563180 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.563137 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-certificates\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:28.563180 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.563184 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-tls\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:28.563635 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.563201 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6653acf2-739a-4628-8f9f-e0658b4a9338-ca-trust-extracted\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:28.563635 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:28.563285 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 05 06:27:28.563635 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:28.563329 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bd7fcc894-s297m: secret "image-registry-tls" not found Sep 05 06:27:28.563635 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.563341 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6653acf2-739a-4628-8f9f-e0658b4a9338-installation-pull-secrets\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:28.563635 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.563369 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-bound-sa-token\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:28.563635 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:28.563416 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-tls podName:6653acf2-739a-4628-8f9f-e0658b4a9338 nodeName:}" failed. No retries permitted until 2026-09-05 06:27:29.063395916 +0000 UTC m=+73.447939039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-tls") pod "image-registry-7bd7fcc894-s297m" (UID: "6653acf2-739a-4628-8f9f-e0658b4a9338") : secret "image-registry-tls" not found Sep 05 06:27:28.563635 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.563438 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6653acf2-739a-4628-8f9f-e0658b4a9338-image-registry-private-configuration\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:28.563635 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.563492 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f99bb\" (UniqueName: \"kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-kube-api-access-f99bb\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:28.563635 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.563537 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6653acf2-739a-4628-8f9f-e0658b4a9338-ca-trust-extracted\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:28.563635 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.563570 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6653acf2-739a-4628-8f9f-e0658b4a9338-trusted-ca\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:28.564045 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.563857 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-certificates\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:28.564392 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.564375 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6653acf2-739a-4628-8f9f-e0658b4a9338-trusted-ca\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:28.565840 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.565823 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6653acf2-739a-4628-8f9f-e0658b4a9338-image-registry-private-configuration\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:28.565886 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.565849 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6653acf2-739a-4628-8f9f-e0658b4a9338-installation-pull-secrets\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:28.572787 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.572766 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-bound-sa-token\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:28.572916 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.572899 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f99bb\" (UniqueName: \"kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-kube-api-access-f99bb\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:28.614537 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.614506 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k68n5"] Sep 05 06:27:28.617471 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:27:28.617444 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62636c45_1709_4102_bbfc_a13e5f699d1e.slice/crio-67935d06f1062d8e3dc3642c4bc83db76496e782195384bf4d891c39eb25ff79 WatchSource:0}: Error finding container 67935d06f1062d8e3dc3642c4bc83db76496e782195384bf4d891c39eb25ff79: Status 404 returned error can't find the container with id 67935d06f1062d8e3dc3642c4bc83db76496e782195384bf4d891c39eb25ff79 Sep 05 06:27:28.670433 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.670383 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-q8tzt" Sep 05 06:27:28.764993 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.764957 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0a59b50-b545-4ea3-ac55-5906910b22c7-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nx5pm\" (UID: \"a0a59b50-b545-4ea3-ac55-5906910b22c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nx5pm" Sep 05 06:27:28.765165 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:28.765105 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 05 06:27:28.765210 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:28.765186 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0a59b50-b545-4ea3-ac55-5906910b22c7-samples-operator-tls podName:a0a59b50-b545-4ea3-ac55-5906910b22c7 nodeName:}" failed. No retries permitted until 2026-09-05 06:27:29.765169486 +0000 UTC m=+74.149712609 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a0a59b50-b545-4ea3-ac55-5906910b22c7-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-nx5pm" (UID: "a0a59b50-b545-4ea3-ac55-5906910b22c7") : secret "samples-operator-tls" not found Sep 05 06:27:28.940078 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:28.939989 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-q8tzt"] Sep 05 06:27:28.945527 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:27:28.945499 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01815101_b04b_43ea_bb6c_bc896fd4b969.slice/crio-0f9f06c4b89a3132ec32bc6cb3d61d83812cca9b58924c3c060694d29f73ac9a WatchSource:0}: Error finding container 0f9f06c4b89a3132ec32bc6cb3d61d83812cca9b58924c3c060694d29f73ac9a: Status 404 returned error can't find the container with id 0f9f06c4b89a3132ec32bc6cb3d61d83812cca9b58924c3c060694d29f73ac9a Sep 05 06:27:29.067242 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:29.067196 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-tls\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:29.067437 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:29.067386 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 05 06:27:29.067437 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:29.067407 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bd7fcc894-s297m: secret "image-registry-tls" not found Sep 05 06:27:29.067525 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:29.067464 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-tls podName:6653acf2-739a-4628-8f9f-e0658b4a9338 nodeName:}" failed. No retries permitted until 2026-09-05 06:27:30.067449183 +0000 UTC m=+74.451992301 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-tls") pod "image-registry-7bd7fcc894-s297m" (UID: "6653acf2-739a-4628-8f9f-e0658b4a9338") : secret "image-registry-tls" not found Sep 05 06:27:29.558943 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:29.558904 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k68n5" event={"ID":"62636c45-1709-4102-bbfc-a13e5f699d1e","Type":"ContainerStarted","Data":"67935d06f1062d8e3dc3642c4bc83db76496e782195384bf4d891c39eb25ff79"} Sep 05 06:27:29.560566 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:29.560536 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-q8tzt" event={"ID":"01815101-b04b-43ea-bb6c-bc896fd4b969","Type":"ContainerStarted","Data":"0f9f06c4b89a3132ec32bc6cb3d61d83812cca9b58924c3c060694d29f73ac9a"} Sep 05 06:27:29.773327 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:29.773251 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0a59b50-b545-4ea3-ac55-5906910b22c7-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nx5pm\" (UID: \"a0a59b50-b545-4ea3-ac55-5906910b22c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nx5pm" Sep 05 06:27:29.773897 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:29.773427 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 05 06:27:29.773897 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:29.773482 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0a59b50-b545-4ea3-ac55-5906910b22c7-samples-operator-tls podName:a0a59b50-b545-4ea3-ac55-5906910b22c7 nodeName:}" failed. No retries permitted until 2026-09-05 06:27:31.773468603 +0000 UTC m=+76.158011705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a0a59b50-b545-4ea3-ac55-5906910b22c7-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-nx5pm" (UID: "a0a59b50-b545-4ea3-ac55-5906910b22c7") : secret "samples-operator-tls" not found Sep 05 06:27:30.075153 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:30.075107 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-tls\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:30.075374 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:30.075279 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 05 06:27:30.075374 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:30.075302 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bd7fcc894-s297m: secret "image-registry-tls" not found Sep 05 06:27:30.075504 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:30.075381 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-tls podName:6653acf2-739a-4628-8f9f-e0658b4a9338 nodeName:}" failed. No retries permitted until 2026-09-05 06:27:32.075360664 +0000 UTC m=+76.459903769 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-tls") pod "image-registry-7bd7fcc894-s297m" (UID: "6653acf2-739a-4628-8f9f-e0658b4a9338") : secret "image-registry-tls" not found Sep 05 06:27:30.565267 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:30.565223 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k68n5" event={"ID":"62636c45-1709-4102-bbfc-a13e5f699d1e","Type":"ContainerStarted","Data":"24d5d75b2496dc4c02b2a36ec4e4cc844ebaa9adf8a1d783cbecf18d167dd90b"} Sep 05 06:27:30.582589 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:30.582540 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k68n5" podStartSLOduration=0.766249397 podStartE2EDuration="2.582524592s" totalImagesPullingTime="1.816275195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:27:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:27:30.582073006 +0000 UTC m=+74.966616130" watchObservedRunningTime="2026-09-05 06:27:30.582524592 +0000 UTC m=+74.967067715" Sep 05 06:27:31.569585 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:31.569541 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-q8tzt" event={"ID":"01815101-b04b-43ea-bb6c-bc896fd4b969","Type":"ContainerStarted","Data":"43d3fd6287bae5db9310d22609a22ca71090b3d176292c92988f944a7bf8855e"} Sep 05 06:27:31.587111 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:31.587055 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-q8tzt" podStartSLOduration=1.156398102 podStartE2EDuration="3.587033283s" totalImagesPullingTime="2.430635181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:27:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:27:31.586668623 +0000 UTC m=+75.971211745" watchObservedRunningTime="2026-09-05 06:27:31.587033283 +0000 UTC m=+75.971576410" Sep 05 06:27:31.790799 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:31.790709 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0a59b50-b545-4ea3-ac55-5906910b22c7-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nx5pm\" (UID: \"a0a59b50-b545-4ea3-ac55-5906910b22c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nx5pm" Sep 05 06:27:31.790960 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:31.790883 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 05 06:27:31.791017 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:31.790970 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0a59b50-b545-4ea3-ac55-5906910b22c7-samples-operator-tls podName:a0a59b50-b545-4ea3-ac55-5906910b22c7 nodeName:}" failed. No retries permitted until 2026-09-05 06:27:35.790950679 +0000 UTC m=+80.175493801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a0a59b50-b545-4ea3-ac55-5906910b22c7-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-nx5pm" (UID: "a0a59b50-b545-4ea3-ac55-5906910b22c7") : secret "samples-operator-tls" not found Sep 05 06:27:32.092708 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:32.092615 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-tls\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:32.092886 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:32.092771 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 05 06:27:32.092886 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:32.092791 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bd7fcc894-s297m: secret "image-registry-tls" not found Sep 05 06:27:32.092886 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:32.092858 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-tls podName:6653acf2-739a-4628-8f9f-e0658b4a9338 nodeName:}" failed. No retries permitted until 2026-09-05 06:27:36.092839656 +0000 UTC m=+80.477382783 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-tls") pod "image-registry-7bd7fcc894-s297m" (UID: "6653acf2-739a-4628-8f9f-e0658b4a9338") : secret "image-registry-tls" not found Sep 05 06:27:32.567140 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:32.567099 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-97l2v"] Sep 05 06:27:32.569255 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:32.569240 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:27:32.571657 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:32.571633 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 05 06:27:32.572015 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:32.571862 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 05 06:27:32.572709 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:32.572687 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 05 06:27:32.572799 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:32.572719 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rb49r\"" Sep 05 06:27:32.572799 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:32.572748 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 05 06:27:32.579625 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:32.579603 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-97l2v"] Sep 05 06:27:32.595754 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:32.595733 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2ecc993-91c0-426b-94e1-664c820c0c57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:27:32.595865 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:32.595763 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2ecc993-91c0-426b-94e1-664c820c0c57-crio-socket\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:27:32.595865 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:32.595788 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbxdh\" (UniqueName: \"kubernetes.io/projected/b2ecc993-91c0-426b-94e1-664c820c0c57-kube-api-access-dbxdh\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:27:32.595962 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:32.595939 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2ecc993-91c0-426b-94e1-664c820c0c57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:27:32.596003 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:32.595988 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2ecc993-91c0-426b-94e1-664c820c0c57-data-volume\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:27:32.696524 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:32.696483 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2ecc993-91c0-426b-94e1-664c820c0c57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:27:32.696524 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:32.696527 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2ecc993-91c0-426b-94e1-664c820c0c57-crio-socket\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:27:32.696788 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:32.696550 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbxdh\" (UniqueName: \"kubernetes.io/projected/b2ecc993-91c0-426b-94e1-664c820c0c57-kube-api-access-dbxdh\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:27:32.696788 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:32.696616 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 05 06:27:32.696788 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:32.696646 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2ecc993-91c0-426b-94e1-664c820c0c57-crio-socket\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:27:32.696788 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:32.696687 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b2ecc993-91c0-426b-94e1-664c820c0c57-kube-rbac-proxy-cm podName:b2ecc993-91c0-426b-94e1-664c820c0c57 nodeName:}" failed. No retries permitted until 2026-09-05 06:27:33.196671037 +0000 UTC m=+77.581214158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b2ecc993-91c0-426b-94e1-664c820c0c57-kube-rbac-proxy-cm") pod "insights-runtime-extractor-97l2v" (UID: "b2ecc993-91c0-426b-94e1-664c820c0c57") : configmap "kube-rbac-proxy" not found Sep 05 06:27:32.696788 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:32.696721 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2ecc993-91c0-426b-94e1-664c820c0c57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:27:32.696788 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:32.696753 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2ecc993-91c0-426b-94e1-664c820c0c57-data-volume\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:27:32.697080 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:32.696835 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 05 06:27:32.697080 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:32.696880 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2ecc993-91c0-426b-94e1-664c820c0c57-insights-runtime-extractor-tls podName:b2ecc993-91c0-426b-94e1-664c820c0c57 nodeName:}" failed. No retries permitted until 2026-09-05 06:27:33.19686798 +0000 UTC m=+77.581411088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b2ecc993-91c0-426b-94e1-664c820c0c57-insights-runtime-extractor-tls") pod "insights-runtime-extractor-97l2v" (UID: "b2ecc993-91c0-426b-94e1-664c820c0c57") : secret "insights-runtime-extractor-tls" not found Sep 05 06:27:32.697179 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:32.697127 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2ecc993-91c0-426b-94e1-664c820c0c57-data-volume\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:27:32.706145 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:32.706115 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbxdh\" (UniqueName: \"kubernetes.io/projected/b2ecc993-91c0-426b-94e1-664c820c0c57-kube-api-access-dbxdh\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:27:33.201386 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:33.201344 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2ecc993-91c0-426b-94e1-664c820c0c57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:27:33.201565 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:33.201442 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 05 06:27:33.201565 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:33.201455 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2ecc993-91c0-426b-94e1-664c820c0c57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:27:33.201565 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:33.201507 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b2ecc993-91c0-426b-94e1-664c820c0c57-kube-rbac-proxy-cm podName:b2ecc993-91c0-426b-94e1-664c820c0c57 nodeName:}" failed. No retries permitted until 2026-09-05 06:27:34.201491165 +0000 UTC m=+78.586034286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b2ecc993-91c0-426b-94e1-664c820c0c57-kube-rbac-proxy-cm") pod "insights-runtime-extractor-97l2v" (UID: "b2ecc993-91c0-426b-94e1-664c820c0c57") : configmap "kube-rbac-proxy" not found Sep 05 06:27:33.201565 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:33.201561 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 05 06:27:33.201703 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:33.201610 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2ecc993-91c0-426b-94e1-664c820c0c57-insights-runtime-extractor-tls podName:b2ecc993-91c0-426b-94e1-664c820c0c57 nodeName:}" failed. No retries permitted until 2026-09-05 06:27:34.201594167 +0000 UTC m=+78.586137270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b2ecc993-91c0-426b-94e1-664c820c0c57-insights-runtime-extractor-tls") pod "insights-runtime-extractor-97l2v" (UID: "b2ecc993-91c0-426b-94e1-664c820c0c57") : secret "insights-runtime-extractor-tls" not found Sep 05 06:27:34.209165 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:34.209124 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2ecc993-91c0-426b-94e1-664c820c0c57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:27:34.209577 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:34.209216 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2ecc993-91c0-426b-94e1-664c820c0c57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:27:34.209577 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:34.209242 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 05 06:27:34.209577 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:34.209330 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b2ecc993-91c0-426b-94e1-664c820c0c57-kube-rbac-proxy-cm podName:b2ecc993-91c0-426b-94e1-664c820c0c57 nodeName:}" failed. No retries permitted until 2026-09-05 06:27:36.209295673 +0000 UTC m=+80.593838788 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b2ecc993-91c0-426b-94e1-664c820c0c57-kube-rbac-proxy-cm") pod "insights-runtime-extractor-97l2v" (UID: "b2ecc993-91c0-426b-94e1-664c820c0c57") : configmap "kube-rbac-proxy" not found Sep 05 06:27:34.209577 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:34.209348 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 05 06:27:34.209577 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:34.209402 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2ecc993-91c0-426b-94e1-664c820c0c57-insights-runtime-extractor-tls podName:b2ecc993-91c0-426b-94e1-664c820c0c57 nodeName:}" failed. No retries permitted until 2026-09-05 06:27:36.20938857 +0000 UTC m=+80.593931685 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b2ecc993-91c0-426b-94e1-664c820c0c57-insights-runtime-extractor-tls") pod "insights-runtime-extractor-97l2v" (UID: "b2ecc993-91c0-426b-94e1-664c820c0c57") : secret "insights-runtime-extractor-tls" not found Sep 05 06:27:34.332614 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:34.332583 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dzlmk_59539926-4278-4795-8ff8-f3482b69df83/dns-node-resolver/0.log" Sep 05 06:27:35.729071 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:35.729046 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m7rvs_60b02f38-b310-4273-bc86-e3eb032c1cd4/node-ca/0.log" Sep 05 06:27:35.821595 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:35.821559 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0a59b50-b545-4ea3-ac55-5906910b22c7-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nx5pm\" (UID: \"a0a59b50-b545-4ea3-ac55-5906910b22c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nx5pm" Sep 05 06:27:35.821760 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:35.821709 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 05 06:27:35.821800 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:35.821775 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0a59b50-b545-4ea3-ac55-5906910b22c7-samples-operator-tls podName:a0a59b50-b545-4ea3-ac55-5906910b22c7 nodeName:}" failed. No retries permitted until 2026-09-05 06:27:43.821758862 +0000 UTC m=+88.206301969 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a0a59b50-b545-4ea3-ac55-5906910b22c7-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-nx5pm" (UID: "a0a59b50-b545-4ea3-ac55-5906910b22c7") : secret "samples-operator-tls" not found Sep 05 06:27:36.123713 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:36.123674 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-tls\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:36.123874 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:36.123832 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 05 06:27:36.123874 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:36.123853 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bd7fcc894-s297m: secret "image-registry-tls" not found Sep 05 06:27:36.123972 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:36.123908 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-tls podName:6653acf2-739a-4628-8f9f-e0658b4a9338 nodeName:}" failed. No retries permitted until 2026-09-05 06:27:44.123893091 +0000 UTC m=+88.508436196 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-tls") pod "image-registry-7bd7fcc894-s297m" (UID: "6653acf2-739a-4628-8f9f-e0658b4a9338") : secret "image-registry-tls" not found Sep 05 06:27:36.224877 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:36.224841 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2ecc993-91c0-426b-94e1-664c820c0c57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:27:36.225089 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:36.224892 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2ecc993-91c0-426b-94e1-664c820c0c57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:27:36.225089 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:36.224974 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 05 06:27:36.225089 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:36.224980 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 05 06:27:36.225089 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:36.225025 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b2ecc993-91c0-426b-94e1-664c820c0c57-kube-rbac-proxy-cm podName:b2ecc993-91c0-426b-94e1-664c820c0c57 nodeName:}" failed. No retries permitted until 2026-09-05 06:27:40.225011826 +0000 UTC m=+84.609554928 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b2ecc993-91c0-426b-94e1-664c820c0c57-kube-rbac-proxy-cm") pod "insights-runtime-extractor-97l2v" (UID: "b2ecc993-91c0-426b-94e1-664c820c0c57") : configmap "kube-rbac-proxy" not found Sep 05 06:27:36.225089 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:36.225038 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2ecc993-91c0-426b-94e1-664c820c0c57-insights-runtime-extractor-tls podName:b2ecc993-91c0-426b-94e1-664c820c0c57 nodeName:}" failed. No retries permitted until 2026-09-05 06:27:40.225032113 +0000 UTC m=+84.609575215 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b2ecc993-91c0-426b-94e1-664c820c0c57-insights-runtime-extractor-tls") pod "insights-runtime-extractor-97l2v" (UID: "b2ecc993-91c0-426b-94e1-664c820c0c57") : secret "insights-runtime-extractor-tls" not found Sep 05 06:27:36.328768 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:36.328735 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cmxtg_ffad8c9d-5409-4c11-a250-686bb69099d4/serve-healthcheck-canary/0.log" Sep 05 06:27:40.256144 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:40.256094 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2ecc993-91c0-426b-94e1-664c820c0c57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:27:40.256639 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:40.256163 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2ecc993-91c0-426b-94e1-664c820c0c57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:27:40.256639 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:40.256268 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 05 06:27:40.256639 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:40.256278 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 05 06:27:40.256639 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:40.256367 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2ecc993-91c0-426b-94e1-664c820c0c57-insights-runtime-extractor-tls podName:b2ecc993-91c0-426b-94e1-664c820c0c57 nodeName:}" failed. No retries permitted until 2026-09-05 06:27:48.256350965 +0000 UTC m=+92.640894085 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b2ecc993-91c0-426b-94e1-664c820c0c57-insights-runtime-extractor-tls") pod "insights-runtime-extractor-97l2v" (UID: "b2ecc993-91c0-426b-94e1-664c820c0c57") : secret "insights-runtime-extractor-tls" not found Sep 05 06:27:40.256639 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:40.256382 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b2ecc993-91c0-426b-94e1-664c820c0c57-kube-rbac-proxy-cm podName:b2ecc993-91c0-426b-94e1-664c820c0c57 nodeName:}" failed. No retries permitted until 2026-09-05 06:27:48.256376382 +0000 UTC m=+92.640919485 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b2ecc993-91c0-426b-94e1-664c820c0c57-kube-rbac-proxy-cm") pod "insights-runtime-extractor-97l2v" (UID: "b2ecc993-91c0-426b-94e1-664c820c0c57") : configmap "kube-rbac-proxy" not found Sep 05 06:27:43.883153 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:43.883105 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0a59b50-b545-4ea3-ac55-5906910b22c7-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nx5pm\" (UID: \"a0a59b50-b545-4ea3-ac55-5906910b22c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nx5pm" Sep 05 06:27:43.885655 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:43.885624 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0a59b50-b545-4ea3-ac55-5906910b22c7-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-nx5pm\" (UID: \"a0a59b50-b545-4ea3-ac55-5906910b22c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nx5pm" Sep 05 06:27:44.061693 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:44.061640 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nx5pm" Sep 05 06:27:44.186619 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:44.186519 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-tls\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:44.189192 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:44.189168 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-tls\") pod \"image-registry-7bd7fcc894-s297m\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:44.275334 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:44.275263 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:44.333513 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:44.333423 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nx5pm"] Sep 05 06:27:44.551873 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:44.551797 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bd7fcc894-s297m"] Sep 05 06:27:44.555798 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:27:44.555772 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6653acf2_739a_4628_8f9f_e0658b4a9338.slice/crio-9dbc62b5f50374e3181f9b9a975223d75d7cf559a0141e2f525b0d7061f104e6 WatchSource:0}: Error finding container 9dbc62b5f50374e3181f9b9a975223d75d7cf559a0141e2f525b0d7061f104e6: Status 404 returned error can't find the container with id 9dbc62b5f50374e3181f9b9a975223d75d7cf559a0141e2f525b0d7061f104e6 Sep 05 06:27:44.602256 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:44.602225 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" event={"ID":"6653acf2-739a-4628-8f9f-e0658b4a9338","Type":"ContainerStarted","Data":"9dbc62b5f50374e3181f9b9a975223d75d7cf559a0141e2f525b0d7061f104e6"} Sep 05 06:27:44.603217 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:44.603186 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nx5pm" event={"ID":"a0a59b50-b545-4ea3-ac55-5906910b22c7","Type":"ContainerStarted","Data":"a152104cce26b5854199df914faa9fed5194076a77213f1ec147313261d803b1"} Sep 05 06:27:45.610153 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:45.610110 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" event={"ID":"6653acf2-739a-4628-8f9f-e0658b4a9338","Type":"ContainerStarted","Data":"a87b9f012d6543f9c1e5e2657aaf5261b8928dc876c1be3f28e8e648b9f7a9c3"} Sep 05 06:27:45.610653 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:45.610241 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:27:45.630809 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:45.630760 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" podStartSLOduration=17.630739959 podStartE2EDuration="17.630739959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:27:45.630564788 +0000 UTC m=+90.015107913" watchObservedRunningTime="2026-09-05 06:27:45.630739959 +0000 UTC m=+90.015283080" Sep 05 06:27:46.614622 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:46.614580 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nx5pm" event={"ID":"a0a59b50-b545-4ea3-ac55-5906910b22c7","Type":"ContainerStarted","Data":"5aabeac7ca7ff311ef935c0390019c30c30d1d63f91f9d106bf9d8e9948bac5d"} Sep 05 06:27:46.615030 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:46.614680 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nx5pm" event={"ID":"a0a59b50-b545-4ea3-ac55-5906910b22c7","Type":"ContainerStarted","Data":"e58923e75ef2c4cbe4ad7660046e47fd984fa16d24cce3f2511c7ee6f6680e14"} Sep 05 06:27:46.632075 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:46.632027 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-nx5pm" podStartSLOduration=17.029616391 podStartE2EDuration="18.632012992s" totalImagesPullingTime="1.602396601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:27:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:27:46.631868781 +0000 UTC m=+91.016411906" watchObservedRunningTime="2026-09-05 06:27:46.632012992 +0000 UTC m=+91.016556115" Sep 05 06:27:48.321790 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:48.321743 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2ecc993-91c0-426b-94e1-664c820c0c57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:27:48.322147 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:48.321827 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2ecc993-91c0-426b-94e1-664c820c0c57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:27:48.322147 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:48.321915 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 05 06:27:48.322147 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:27:48.321967 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2ecc993-91c0-426b-94e1-664c820c0c57-insights-runtime-extractor-tls podName:b2ecc993-91c0-426b-94e1-664c820c0c57 nodeName:}" failed. No retries permitted until 2026-09-05 06:28:04.321954572 +0000 UTC m=+108.706497677 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b2ecc993-91c0-426b-94e1-664c820c0c57-insights-runtime-extractor-tls") pod "insights-runtime-extractor-97l2v" (UID: "b2ecc993-91c0-426b-94e1-664c820c0c57") : secret "insights-runtime-extractor-tls" not found Sep 05 06:27:48.322266 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:48.322236 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2ecc993-91c0-426b-94e1-664c820c0c57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:27:54.872135 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:54.872099 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f8f4891-a720-43c1-bdf3-73b8250fde4f-metrics-tls\") pod \"dns-default-j4dq5\" (UID: \"3f8f4891-a720-43c1-bdf3-73b8250fde4f\") " pod="openshift-dns/dns-default-j4dq5" Sep 05 06:27:54.874552 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:54.874526 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f8f4891-a720-43c1-bdf3-73b8250fde4f-metrics-tls\") pod \"dns-default-j4dq5\" (UID: \"3f8f4891-a720-43c1-bdf3-73b8250fde4f\") " pod="openshift-dns/dns-default-j4dq5" Sep 05 06:27:54.888405 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:54.888382 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v76tl\"" Sep 05 06:27:54.896550 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:54.896533 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j4dq5" Sep 05 06:27:55.170391 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:55.170301 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j4dq5"] Sep 05 06:27:55.174512 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:27:55.174483 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f8f4891_a720_43c1_bdf3_73b8250fde4f.slice/crio-ad732fe6d04c64e0e580a4123161911f519983bb52b52c12e8a0b7a4b0907bfc WatchSource:0}: Error finding container ad732fe6d04c64e0e580a4123161911f519983bb52b52c12e8a0b7a4b0907bfc: Status 404 returned error can't find the container with id ad732fe6d04c64e0e580a4123161911f519983bb52b52c12e8a0b7a4b0907bfc Sep 05 06:27:55.643404 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:55.643368 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j4dq5" event={"ID":"3f8f4891-a720-43c1-bdf3-73b8250fde4f","Type":"ContainerStarted","Data":"ad732fe6d04c64e0e580a4123161911f519983bb52b52c12e8a0b7a4b0907bfc"} Sep 05 06:27:56.909328 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:56.909289 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-92gjz"] Sep 05 06:27:56.912505 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:56.912488 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-92gjz" Sep 05 06:27:56.916043 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:56.916022 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 05 06:27:56.916194 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:56.916055 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 05 06:27:56.916194 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:56.916021 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-nw6lg\"" Sep 05 06:27:56.918894 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:56.918872 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8656cd9cfb-2sg9n"] Sep 05 06:27:56.922692 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:56.922673 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-92gjz"] Sep 05 06:27:56.922785 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:56.922760 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8656cd9cfb-2sg9n" Sep 05 06:27:56.925952 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:56.925923 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 05 06:27:56.926072 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:56.926001 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 05 06:27:56.926635 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:56.926611 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 05 06:27:56.926803 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:56.926780 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 05 06:27:56.935242 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:56.935212 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8656cd9cfb-2sg9n"] Sep 05 06:27:56.979681 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:56.979643 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bd7fcc894-s297m"] Sep 05 06:27:56.988438 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:56.988410 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c1889ac-a439-4df5-8e29-161a26c6d05b-tmp\") pod \"klusterlet-addon-workmgr-8656cd9cfb-2sg9n\" (UID: \"8c1889ac-a439-4df5-8e29-161a26c6d05b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8656cd9cfb-2sg9n" Sep 05 06:27:56.988581 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:56.988449 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv4bk\" (UniqueName: \"kubernetes.io/projected/8c1889ac-a439-4df5-8e29-161a26c6d05b-kube-api-access-qv4bk\") pod \"klusterlet-addon-workmgr-8656cd9cfb-2sg9n\" (UID: \"8c1889ac-a439-4df5-8e29-161a26c6d05b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8656cd9cfb-2sg9n" Sep 05 06:27:56.988636 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:56.988591 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1185aa3e-9993-48a2-8107-c13c92815d36-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-92gjz\" (UID: \"1185aa3e-9993-48a2-8107-c13c92815d36\") " pod="openshift-network-console/networking-console-plugin-79977f9844-92gjz" Sep 05 06:27:56.988636 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:56.988623 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1185aa3e-9993-48a2-8107-c13c92815d36-nginx-conf\") pod \"networking-console-plugin-79977f9844-92gjz\" (UID: \"1185aa3e-9993-48a2-8107-c13c92815d36\") " pod="openshift-network-console/networking-console-plugin-79977f9844-92gjz" Sep 05 06:27:56.988721 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:56.988645 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8c1889ac-a439-4df5-8e29-161a26c6d05b-klusterlet-config\") pod \"klusterlet-addon-workmgr-8656cd9cfb-2sg9n\" (UID: \"8c1889ac-a439-4df5-8e29-161a26c6d05b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8656cd9cfb-2sg9n" Sep 05 06:27:57.015613 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.015581 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7759f7f78b-bvxqx"] Sep 05 06:27:57.037581 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.037543 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7759f7f78b-bvxqx"] Sep 05 06:27:57.037733 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.037687 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.089744 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.089710 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70b7f437-85ac-49f8-bb86-1266ac5db0b6-installation-pull-secrets\") pod \"image-registry-7759f7f78b-bvxqx\" (UID: \"70b7f437-85ac-49f8-bb86-1266ac5db0b6\") " pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.089744 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.089743 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70b7f437-85ac-49f8-bb86-1266ac5db0b6-bound-sa-token\") pod \"image-registry-7759f7f78b-bvxqx\" (UID: \"70b7f437-85ac-49f8-bb86-1266ac5db0b6\") " pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.089923 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.089763 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70b7f437-85ac-49f8-bb86-1266ac5db0b6-trusted-ca\") pod \"image-registry-7759f7f78b-bvxqx\" (UID: \"70b7f437-85ac-49f8-bb86-1266ac5db0b6\") " pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.089923 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.089839 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1185aa3e-9993-48a2-8107-c13c92815d36-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-92gjz\" (UID: \"1185aa3e-9993-48a2-8107-c13c92815d36\") " pod="openshift-network-console/networking-console-plugin-79977f9844-92gjz" Sep 05 06:27:57.089923 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.089862 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b7f437-85ac-49f8-bb86-1266ac5db0b6-registry-tls\") pod \"image-registry-7759f7f78b-bvxqx\" (UID: \"70b7f437-85ac-49f8-bb86-1266ac5db0b6\") " pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.089923 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.089878 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1185aa3e-9993-48a2-8107-c13c92815d36-nginx-conf\") pod \"networking-console-plugin-79977f9844-92gjz\" (UID: \"1185aa3e-9993-48a2-8107-c13c92815d36\") " pod="openshift-network-console/networking-console-plugin-79977f9844-92gjz" Sep 05 06:27:57.089923 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.089897 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8c1889ac-a439-4df5-8e29-161a26c6d05b-klusterlet-config\") pod \"klusterlet-addon-workmgr-8656cd9cfb-2sg9n\" (UID: \"8c1889ac-a439-4df5-8e29-161a26c6d05b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8656cd9cfb-2sg9n" Sep 05 06:27:57.090142 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.089960 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c1889ac-a439-4df5-8e29-161a26c6d05b-tmp\") pod \"klusterlet-addon-workmgr-8656cd9cfb-2sg9n\" (UID: \"8c1889ac-a439-4df5-8e29-161a26c6d05b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8656cd9cfb-2sg9n" Sep 05 06:27:57.090142 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.089986 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv4bk\" (UniqueName: \"kubernetes.io/projected/8c1889ac-a439-4df5-8e29-161a26c6d05b-kube-api-access-qv4bk\") pod \"klusterlet-addon-workmgr-8656cd9cfb-2sg9n\" (UID: \"8c1889ac-a439-4df5-8e29-161a26c6d05b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8656cd9cfb-2sg9n" Sep 05 06:27:57.090142 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.090024 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70b7f437-85ac-49f8-bb86-1266ac5db0b6-ca-trust-extracted\") pod \"image-registry-7759f7f78b-bvxqx\" (UID: \"70b7f437-85ac-49f8-bb86-1266ac5db0b6\") " pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.090142 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.090052 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70b7f437-85ac-49f8-bb86-1266ac5db0b6-image-registry-private-configuration\") pod \"image-registry-7759f7f78b-bvxqx\" (UID: \"70b7f437-85ac-49f8-bb86-1266ac5db0b6\") " pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.090142 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.090118 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70b7f437-85ac-49f8-bb86-1266ac5db0b6-registry-certificates\") pod \"image-registry-7759f7f78b-bvxqx\" (UID: \"70b7f437-85ac-49f8-bb86-1266ac5db0b6\") " pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.090379 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.090172 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn8rw\" (UniqueName: \"kubernetes.io/projected/70b7f437-85ac-49f8-bb86-1266ac5db0b6-kube-api-access-bn8rw\") pod \"image-registry-7759f7f78b-bvxqx\" (UID: \"70b7f437-85ac-49f8-bb86-1266ac5db0b6\") " pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.090468 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.090449 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c1889ac-a439-4df5-8e29-161a26c6d05b-tmp\") pod \"klusterlet-addon-workmgr-8656cd9cfb-2sg9n\" (UID: \"8c1889ac-a439-4df5-8e29-161a26c6d05b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8656cd9cfb-2sg9n" Sep 05 06:27:57.090629 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.090612 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1185aa3e-9993-48a2-8107-c13c92815d36-nginx-conf\") pod \"networking-console-plugin-79977f9844-92gjz\" (UID: \"1185aa3e-9993-48a2-8107-c13c92815d36\") " pod="openshift-network-console/networking-console-plugin-79977f9844-92gjz" Sep 05 06:27:57.092643 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.092616 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1185aa3e-9993-48a2-8107-c13c92815d36-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-92gjz\" (UID: \"1185aa3e-9993-48a2-8107-c13c92815d36\") " pod="openshift-network-console/networking-console-plugin-79977f9844-92gjz" Sep 05 06:27:57.092748 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.092615 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8c1889ac-a439-4df5-8e29-161a26c6d05b-klusterlet-config\") pod \"klusterlet-addon-workmgr-8656cd9cfb-2sg9n\" (UID: \"8c1889ac-a439-4df5-8e29-161a26c6d05b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8656cd9cfb-2sg9n" Sep 05 06:27:57.102403 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.102380 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv4bk\" (UniqueName: \"kubernetes.io/projected/8c1889ac-a439-4df5-8e29-161a26c6d05b-kube-api-access-qv4bk\") pod \"klusterlet-addon-workmgr-8656cd9cfb-2sg9n\" (UID: \"8c1889ac-a439-4df5-8e29-161a26c6d05b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8656cd9cfb-2sg9n" Sep 05 06:27:57.191338 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.191239 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn8rw\" (UniqueName: \"kubernetes.io/projected/70b7f437-85ac-49f8-bb86-1266ac5db0b6-kube-api-access-bn8rw\") pod \"image-registry-7759f7f78b-bvxqx\" (UID: \"70b7f437-85ac-49f8-bb86-1266ac5db0b6\") " pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.191338 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.191277 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70b7f437-85ac-49f8-bb86-1266ac5db0b6-installation-pull-secrets\") pod \"image-registry-7759f7f78b-bvxqx\" (UID: \"70b7f437-85ac-49f8-bb86-1266ac5db0b6\") " pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.191338 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.191295 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70b7f437-85ac-49f8-bb86-1266ac5db0b6-bound-sa-token\") pod \"image-registry-7759f7f78b-bvxqx\" (UID: \"70b7f437-85ac-49f8-bb86-1266ac5db0b6\") " pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.191542 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.191449 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70b7f437-85ac-49f8-bb86-1266ac5db0b6-trusted-ca\") pod \"image-registry-7759f7f78b-bvxqx\" (UID: \"70b7f437-85ac-49f8-bb86-1266ac5db0b6\") " pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.191542 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.191513 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b7f437-85ac-49f8-bb86-1266ac5db0b6-registry-tls\") pod \"image-registry-7759f7f78b-bvxqx\" (UID: \"70b7f437-85ac-49f8-bb86-1266ac5db0b6\") " pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.191607 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.191566 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70b7f437-85ac-49f8-bb86-1266ac5db0b6-ca-trust-extracted\") pod \"image-registry-7759f7f78b-bvxqx\" (UID: \"70b7f437-85ac-49f8-bb86-1266ac5db0b6\") " pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.191607 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.191586 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70b7f437-85ac-49f8-bb86-1266ac5db0b6-image-registry-private-configuration\") pod \"image-registry-7759f7f78b-bvxqx\" (UID: \"70b7f437-85ac-49f8-bb86-1266ac5db0b6\") " pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.191682 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.191605 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70b7f437-85ac-49f8-bb86-1266ac5db0b6-registry-certificates\") pod \"image-registry-7759f7f78b-bvxqx\" (UID: \"70b7f437-85ac-49f8-bb86-1266ac5db0b6\") " pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.192021 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.191990 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70b7f437-85ac-49f8-bb86-1266ac5db0b6-ca-trust-extracted\") pod \"image-registry-7759f7f78b-bvxqx\" (UID: \"70b7f437-85ac-49f8-bb86-1266ac5db0b6\") " pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.192353 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.192333 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70b7f437-85ac-49f8-bb86-1266ac5db0b6-registry-certificates\") pod \"image-registry-7759f7f78b-bvxqx\" (UID: \"70b7f437-85ac-49f8-bb86-1266ac5db0b6\") " pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.192876 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.192853 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70b7f437-85ac-49f8-bb86-1266ac5db0b6-trusted-ca\") pod \"image-registry-7759f7f78b-bvxqx\" (UID: \"70b7f437-85ac-49f8-bb86-1266ac5db0b6\") " pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.193885 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.193864 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70b7f437-85ac-49f8-bb86-1266ac5db0b6-image-registry-private-configuration\") pod \"image-registry-7759f7f78b-bvxqx\" (UID: \"70b7f437-85ac-49f8-bb86-1266ac5db0b6\") " pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.193967 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.193872 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70b7f437-85ac-49f8-bb86-1266ac5db0b6-installation-pull-secrets\") pod \"image-registry-7759f7f78b-bvxqx\" (UID: \"70b7f437-85ac-49f8-bb86-1266ac5db0b6\") " pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.194191 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.194175 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b7f437-85ac-49f8-bb86-1266ac5db0b6-registry-tls\") pod \"image-registry-7759f7f78b-bvxqx\" (UID: \"70b7f437-85ac-49f8-bb86-1266ac5db0b6\") " pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.202347 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.202301 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn8rw\" (UniqueName: \"kubernetes.io/projected/70b7f437-85ac-49f8-bb86-1266ac5db0b6-kube-api-access-bn8rw\") pod \"image-registry-7759f7f78b-bvxqx\" (UID: \"70b7f437-85ac-49f8-bb86-1266ac5db0b6\") " pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.202529 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.202511 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70b7f437-85ac-49f8-bb86-1266ac5db0b6-bound-sa-token\") pod \"image-registry-7759f7f78b-bvxqx\" (UID: \"70b7f437-85ac-49f8-bb86-1266ac5db0b6\") " pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.223708 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.223682 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-92gjz" Sep 05 06:27:57.237900 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.237874 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8656cd9cfb-2sg9n" Sep 05 06:27:57.347751 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.347725 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:57.535242 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.535162 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-92gjz"] Sep 05 06:27:57.539676 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:27:57.539650 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1185aa3e_9993_48a2_8107_c13c92815d36.slice/crio-68274ba2ce13312041a926f49c5e140fa405b12a0836e3d9b449fe7c82e9bd0f WatchSource:0}: Error finding container 68274ba2ce13312041a926f49c5e140fa405b12a0836e3d9b449fe7c82e9bd0f: Status 404 returned error can't find the container with id 68274ba2ce13312041a926f49c5e140fa405b12a0836e3d9b449fe7c82e9bd0f Sep 05 06:27:57.579037 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.578998 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8656cd9cfb-2sg9n"] Sep 05 06:27:57.581593 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:27:57.581562 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1889ac_a439_4df5_8e29_161a26c6d05b.slice/crio-86df1c2477ba996652808d78826fd519eba61a1bba5a25492863bf16cc088de9 WatchSource:0}: Error finding container 86df1c2477ba996652808d78826fd519eba61a1bba5a25492863bf16cc088de9: Status 404 returned error can't find the container with id 86df1c2477ba996652808d78826fd519eba61a1bba5a25492863bf16cc088de9 Sep 05 06:27:57.652143 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.652109 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j4dq5" event={"ID":"3f8f4891-a720-43c1-bdf3-73b8250fde4f","Type":"ContainerStarted","Data":"312a9e05d38cdb8ed6b1f052f5cc3a6e7dcfeccbd8223b05ff661fc08b179c54"} Sep 05 06:27:57.652304 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.652175 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-j4dq5" Sep 05 06:27:57.652304 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.652200 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j4dq5" event={"ID":"3f8f4891-a720-43c1-bdf3-73b8250fde4f","Type":"ContainerStarted","Data":"f12fbdfa64c382722f55dd70d758c073565fd561a8b5e89e2d5e5ff848291d68"} Sep 05 06:27:57.653647 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.653616 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8656cd9cfb-2sg9n" event={"ID":"8c1889ac-a439-4df5-8e29-161a26c6d05b","Type":"ContainerStarted","Data":"86df1c2477ba996652808d78826fd519eba61a1bba5a25492863bf16cc088de9"} Sep 05 06:27:57.654597 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.654573 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7759f7f78b-bvxqx"] Sep 05 06:27:57.654907 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.654890 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-92gjz" event={"ID":"1185aa3e-9993-48a2-8107-c13c92815d36","Type":"ContainerStarted","Data":"68274ba2ce13312041a926f49c5e140fa405b12a0836e3d9b449fe7c82e9bd0f"} Sep 05 06:27:57.657640 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:27:57.657619 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b7f437_85ac_49f8_bb86_1266ac5db0b6.slice/crio-8f452bbbb3fe53f2228555eed272db5e6acfd9885000c568343599939b564c13 WatchSource:0}: Error finding container 8f452bbbb3fe53f2228555eed272db5e6acfd9885000c568343599939b564c13: Status 404 returned error can't find the container with id 8f452bbbb3fe53f2228555eed272db5e6acfd9885000c568343599939b564c13 Sep 05 06:27:57.674049 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:57.674011 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-j4dq5" podStartSLOduration=66.261257655 podStartE2EDuration="1m7.67399674s" totalImagesPullingTime="1.412739085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:26:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:27:57.672996493 +0000 UTC m=+102.057539629" watchObservedRunningTime="2026-09-05 06:27:57.67399674 +0000 UTC m=+102.058539863" Sep 05 06:27:58.556675 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:58.556634 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nccpk" Sep 05 06:27:58.664994 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:58.664959 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" event={"ID":"70b7f437-85ac-49f8-bb86-1266ac5db0b6","Type":"ContainerStarted","Data":"b0243f6ec68568e1e893f9b6f8d2c886a4fa625c79aa5f56e9ec35ca71badd88"} Sep 05 06:27:58.665209 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:58.665010 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" event={"ID":"70b7f437-85ac-49f8-bb86-1266ac5db0b6","Type":"ContainerStarted","Data":"8f452bbbb3fe53f2228555eed272db5e6acfd9885000c568343599939b564c13"} Sep 05 06:27:58.665209 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:58.665055 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:27:58.691959 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:27:58.690974 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" podStartSLOduration=2.690951987 podStartE2EDuration="2.690951987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:27:58.689542018 +0000 UTC m=+103.074085142" watchObservedRunningTime="2026-09-05 06:27:58.690951987 +0000 UTC m=+103.075495092" Sep 05 06:28:01.679120 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:01.679087 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-92gjz" event={"ID":"1185aa3e-9993-48a2-8107-c13c92815d36","Type":"ContainerStarted","Data":"29d88a2e1295ca1da34c4c4c3413dce4d07fe76186fc777ba89255d1bf83fd57"} Sep 05 06:28:01.697185 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:01.697118 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-92gjz" podStartSLOduration=1.925743841 podStartE2EDuration="5.697097938s" totalImagesPullingTime="3.771354097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:27:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:28:01.696150788 +0000 UTC m=+106.080693912" watchObservedRunningTime="2026-09-05 06:28:01.697097938 +0000 UTC m=+106.081641064" Sep 05 06:28:02.684266 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:02.684230 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8656cd9cfb-2sg9n" event={"ID":"8c1889ac-a439-4df5-8e29-161a26c6d05b","Type":"ContainerStarted","Data":"669afb0436d4fbdf8d11495ca13d7f4ef895c1a828860d0cd669a9c19d537f71"} Sep 05 06:28:02.684704 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:02.684580 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8656cd9cfb-2sg9n" Sep 05 06:28:02.686053 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:02.686034 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8656cd9cfb-2sg9n" Sep 05 06:28:02.702630 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:02.702587 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8656cd9cfb-2sg9n" podStartSLOduration=2.680880154 podStartE2EDuration="6.702572626s" totalImagesPullingTime="4.021692472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:27:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:28:02.701759371 +0000 UTC m=+107.086302496" watchObservedRunningTime="2026-09-05 06:28:02.702572626 +0000 UTC m=+107.087115750" Sep 05 06:28:04.354817 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:04.354757 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2ecc993-91c0-426b-94e1-664c820c0c57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:28:04.357300 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:04.357276 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2ecc993-91c0-426b-94e1-664c820c0c57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-97l2v\" (UID: \"b2ecc993-91c0-426b-94e1-664c820c0c57\") " pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:28:04.378250 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:04.378225 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-97l2v" Sep 05 06:28:04.652814 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:04.652731 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-97l2v"] Sep 05 06:28:04.657236 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:28:04.657207 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2ecc993_91c0_426b_94e1_664c820c0c57.slice/crio-b44abe1e0d2d67d95ea7e4645710fcae71cc1c3070725ba70156fa78ca885745 WatchSource:0}: Error finding container b44abe1e0d2d67d95ea7e4645710fcae71cc1c3070725ba70156fa78ca885745: Status 404 returned error can't find the container with id b44abe1e0d2d67d95ea7e4645710fcae71cc1c3070725ba70156fa78ca885745 Sep 05 06:28:04.691187 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:04.691154 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-97l2v" event={"ID":"b2ecc993-91c0-426b-94e1-664c820c0c57","Type":"ContainerStarted","Data":"b44abe1e0d2d67d95ea7e4645710fcae71cc1c3070725ba70156fa78ca885745"} Sep 05 06:28:05.696770 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:05.696737 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-97l2v" event={"ID":"b2ecc993-91c0-426b-94e1-664c820c0c57","Type":"ContainerStarted","Data":"c29473351090f70f6df495b743ba5e32d8ff50bae2f723a8fba1bc4430fd90e4"} Sep 05 06:28:05.697126 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:05.696793 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-97l2v" event={"ID":"b2ecc993-91c0-426b-94e1-664c820c0c57","Type":"ContainerStarted","Data":"6a4e9aeb88e08e93657752e8936fb8e9f3368e095b398a74a18e75ce2b6982ad"} Sep 05 06:28:06.985536 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:06.985500 2565 patch_prober.go:28] interesting pod/image-registry-7bd7fcc894-s297m container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 05 06:28:06.985900 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:06.985557 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" podUID="6653acf2-739a-4628-8f9f-e0658b4a9338" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 05 06:28:07.666103 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:07.666073 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-j4dq5" Sep 05 06:28:07.708942 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:07.708906 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-97l2v" event={"ID":"b2ecc993-91c0-426b-94e1-664c820c0c57","Type":"ContainerStarted","Data":"eff1da5185b53576ea819bdd9f7f33ad6c20afa3be19bfaf6f50356bc30be2c3"} Sep 05 06:28:07.729828 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:07.729782 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-97l2v" podStartSLOduration=33.640567262 podStartE2EDuration="35.729765552s" totalImagesPullingTime="2.08919829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 06:27:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:28:07.728180546 +0000 UTC m=+112.112723671" watchObservedRunningTime="2026-09-05 06:28:07.729765552 +0000 UTC m=+112.114308672" Sep 05 06:28:10.302819 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.302783 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-pxvmp"] Sep 05 06:28:10.305198 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.305180 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pxvmp" Sep 05 06:28:10.307172 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.307145 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 05 06:28:10.307914 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.307897 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 05 06:28:10.308830 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.308807 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 05 06:28:10.308916 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.308834 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 05 06:28:10.308916 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.308815 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 05 06:28:10.308916 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.308910 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4pf7n\"" Sep 05 06:28:10.321678 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.321648 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb"] Sep 05 06:28:10.323815 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.323796 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-pxvmp"] Sep 05 06:28:10.323908 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.323888 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb" Sep 05 06:28:10.326228 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.326208 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 05 06:28:10.326336 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.326246 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rvrrc\"" Sep 05 06:28:10.326411 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.326379 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 05 06:28:10.326488 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.326473 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 05 06:28:10.330583 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.330565 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-v57dr"] Sep 05 06:28:10.332777 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.332762 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.335709 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.335692 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 05 06:28:10.336254 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.336234 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 05 06:28:10.336254 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.336246 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bcsr2\"" Sep 05 06:28:10.336436 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.336371 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb"] Sep 05 06:28:10.336512 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.336472 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 05 06:28:10.403907 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.403874 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03b45c53-3ea5-4817-b164-413cf5f2ae5d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-pxvmp\" (UID: \"03b45c53-3ea5-4817-b164-413cf5f2ae5d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pxvmp" Sep 05 06:28:10.403907 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.403907 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03b45c53-3ea5-4817-b164-413cf5f2ae5d-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-pxvmp\" (UID: \"03b45c53-3ea5-4817-b164-413cf5f2ae5d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pxvmp" Sep 05 06:28:10.404120 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.403925 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prnxs\" (UniqueName: \"kubernetes.io/projected/5ac12b19-6ff2-41f9-a7d7-e4416e7962a6-kube-api-access-prnxs\") pod \"kube-state-metrics-74cb57df7-zwhpb\" (UID: \"5ac12b19-6ff2-41f9-a7d7-e4416e7962a6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb" Sep 05 06:28:10.404120 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.403952 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ac12b19-6ff2-41f9-a7d7-e4416e7962a6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-zwhpb\" (UID: \"5ac12b19-6ff2-41f9-a7d7-e4416e7962a6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb" Sep 05 06:28:10.404120 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.404017 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ac12b19-6ff2-41f9-a7d7-e4416e7962a6-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-zwhpb\" (UID: \"5ac12b19-6ff2-41f9-a7d7-e4416e7962a6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb" Sep 05 06:28:10.404120 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.404059 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5ac12b19-6ff2-41f9-a7d7-e4416e7962a6-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-zwhpb\" (UID: \"5ac12b19-6ff2-41f9-a7d7-e4416e7962a6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb" Sep 05 06:28:10.404120 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.404085 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/03b45c53-3ea5-4817-b164-413cf5f2ae5d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-pxvmp\" (UID: \"03b45c53-3ea5-4817-b164-413cf5f2ae5d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pxvmp" Sep 05 06:28:10.404120 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.404116 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ac12b19-6ff2-41f9-a7d7-e4416e7962a6-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-zwhpb\" (UID: \"5ac12b19-6ff2-41f9-a7d7-e4416e7962a6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb" Sep 05 06:28:10.404361 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.404133 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f64vq\" (UniqueName: \"kubernetes.io/projected/03b45c53-3ea5-4817-b164-413cf5f2ae5d-kube-api-access-f64vq\") pod \"openshift-state-metrics-657fd6c54-pxvmp\" (UID: \"03b45c53-3ea5-4817-b164-413cf5f2ae5d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pxvmp" Sep 05 06:28:10.404361 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.404157 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5ac12b19-6ff2-41f9-a7d7-e4416e7962a6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-zwhpb\" (UID: \"5ac12b19-6ff2-41f9-a7d7-e4416e7962a6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb" Sep 05 06:28:10.504551 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.504516 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8cb7ae97-6473-478e-b8e9-534be9f60b02-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.504551 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.504553 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ac12b19-6ff2-41f9-a7d7-e4416e7962a6-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-zwhpb\" (UID: \"5ac12b19-6ff2-41f9-a7d7-e4416e7962a6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb" Sep 05 06:28:10.504763 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.504575 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8cb7ae97-6473-478e-b8e9-534be9f60b02-node-exporter-accelerators-collector-config\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.504763 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.504608 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5ac12b19-6ff2-41f9-a7d7-e4416e7962a6-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-zwhpb\" (UID: \"5ac12b19-6ff2-41f9-a7d7-e4416e7962a6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb" Sep 05 06:28:10.504763 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.504631 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8cb7ae97-6473-478e-b8e9-534be9f60b02-node-exporter-tls\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.504763 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.504664 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/03b45c53-3ea5-4817-b164-413cf5f2ae5d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-pxvmp\" (UID: \"03b45c53-3ea5-4817-b164-413cf5f2ae5d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pxvmp" Sep 05 06:28:10.504763 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.504690 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cb7ae97-6473-478e-b8e9-534be9f60b02-sys\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.504763 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.504745 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cb7ae97-6473-478e-b8e9-534be9f60b02-metrics-client-ca\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.505009 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.504766 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ac12b19-6ff2-41f9-a7d7-e4416e7962a6-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-zwhpb\" (UID: \"5ac12b19-6ff2-41f9-a7d7-e4416e7962a6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb" Sep 05 06:28:10.505009 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.504788 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f64vq\" (UniqueName: \"kubernetes.io/projected/03b45c53-3ea5-4817-b164-413cf5f2ae5d-kube-api-access-f64vq\") pod \"openshift-state-metrics-657fd6c54-pxvmp\" (UID: \"03b45c53-3ea5-4817-b164-413cf5f2ae5d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pxvmp" Sep 05 06:28:10.505009 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.504808 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsxjb\" (UniqueName: \"kubernetes.io/projected/8cb7ae97-6473-478e-b8e9-534be9f60b02-kube-api-access-vsxjb\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.505009 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.504837 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8cb7ae97-6473-478e-b8e9-534be9f60b02-node-exporter-wtmp\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.505009 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.504979 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5ac12b19-6ff2-41f9-a7d7-e4416e7962a6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-zwhpb\" (UID: \"5ac12b19-6ff2-41f9-a7d7-e4416e7962a6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb" Sep 05 06:28:10.505247 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.504993 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5ac12b19-6ff2-41f9-a7d7-e4416e7962a6-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-zwhpb\" (UID: \"5ac12b19-6ff2-41f9-a7d7-e4416e7962a6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb" Sep 05 06:28:10.505247 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.505032 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8cb7ae97-6473-478e-b8e9-534be9f60b02-root\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.505247 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.505061 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8cb7ae97-6473-478e-b8e9-534be9f60b02-node-exporter-textfile\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.505247 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.505112 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03b45c53-3ea5-4817-b164-413cf5f2ae5d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-pxvmp\" (UID: \"03b45c53-3ea5-4817-b164-413cf5f2ae5d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pxvmp" Sep 05 06:28:10.505247 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.505141 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03b45c53-3ea5-4817-b164-413cf5f2ae5d-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-pxvmp\" (UID: \"03b45c53-3ea5-4817-b164-413cf5f2ae5d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pxvmp" Sep 05 06:28:10.505247 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.505171 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prnxs\" (UniqueName: \"kubernetes.io/projected/5ac12b19-6ff2-41f9-a7d7-e4416e7962a6-kube-api-access-prnxs\") pod \"kube-state-metrics-74cb57df7-zwhpb\" (UID: \"5ac12b19-6ff2-41f9-a7d7-e4416e7962a6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb" Sep 05 06:28:10.505247 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.505212 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ac12b19-6ff2-41f9-a7d7-e4416e7962a6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-zwhpb\" (UID: \"5ac12b19-6ff2-41f9-a7d7-e4416e7962a6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb" Sep 05 06:28:10.505661 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.505576 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ac12b19-6ff2-41f9-a7d7-e4416e7962a6-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-zwhpb\" (UID: \"5ac12b19-6ff2-41f9-a7d7-e4416e7962a6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb" Sep 05 06:28:10.505661 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.505650 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5ac12b19-6ff2-41f9-a7d7-e4416e7962a6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-zwhpb\" (UID: \"5ac12b19-6ff2-41f9-a7d7-e4416e7962a6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb" Sep 05 06:28:10.506039 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.506017 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03b45c53-3ea5-4817-b164-413cf5f2ae5d-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-pxvmp\" (UID: \"03b45c53-3ea5-4817-b164-413cf5f2ae5d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pxvmp" Sep 05 06:28:10.507543 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.507521 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03b45c53-3ea5-4817-b164-413cf5f2ae5d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-pxvmp\" (UID: \"03b45c53-3ea5-4817-b164-413cf5f2ae5d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pxvmp" Sep 05 06:28:10.507543 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.507529 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ac12b19-6ff2-41f9-a7d7-e4416e7962a6-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-zwhpb\" (UID: \"5ac12b19-6ff2-41f9-a7d7-e4416e7962a6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb" Sep 05 06:28:10.507543 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.507529 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/03b45c53-3ea5-4817-b164-413cf5f2ae5d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-pxvmp\" (UID: \"03b45c53-3ea5-4817-b164-413cf5f2ae5d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pxvmp" Sep 05 06:28:10.507814 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.507795 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ac12b19-6ff2-41f9-a7d7-e4416e7962a6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-zwhpb\" (UID: \"5ac12b19-6ff2-41f9-a7d7-e4416e7962a6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb" Sep 05 06:28:10.513634 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.513610 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f64vq\" (UniqueName: \"kubernetes.io/projected/03b45c53-3ea5-4817-b164-413cf5f2ae5d-kube-api-access-f64vq\") pod \"openshift-state-metrics-657fd6c54-pxvmp\" (UID: \"03b45c53-3ea5-4817-b164-413cf5f2ae5d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pxvmp" Sep 05 06:28:10.513969 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.513944 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prnxs\" (UniqueName: \"kubernetes.io/projected/5ac12b19-6ff2-41f9-a7d7-e4416e7962a6-kube-api-access-prnxs\") pod \"kube-state-metrics-74cb57df7-zwhpb\" (UID: \"5ac12b19-6ff2-41f9-a7d7-e4416e7962a6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb" Sep 05 06:28:10.605691 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.605667 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8cb7ae97-6473-478e-b8e9-534be9f60b02-node-exporter-accelerators-collector-config\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.605867 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.605702 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8cb7ae97-6473-478e-b8e9-534be9f60b02-node-exporter-tls\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.605867 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.605726 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cb7ae97-6473-478e-b8e9-534be9f60b02-sys\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.605867 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.605754 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cb7ae97-6473-478e-b8e9-534be9f60b02-metrics-client-ca\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.605867 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.605780 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsxjb\" (UniqueName: \"kubernetes.io/projected/8cb7ae97-6473-478e-b8e9-534be9f60b02-kube-api-access-vsxjb\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.605867 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.605808 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8cb7ae97-6473-478e-b8e9-534be9f60b02-node-exporter-wtmp\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.605867 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.605836 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8cb7ae97-6473-478e-b8e9-534be9f60b02-root\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.605867 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.605860 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8cb7ae97-6473-478e-b8e9-534be9f60b02-node-exporter-textfile\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.606196 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.605807 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cb7ae97-6473-478e-b8e9-534be9f60b02-sys\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.606196 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.605911 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8cb7ae97-6473-478e-b8e9-534be9f60b02-node-exporter-wtmp\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.606196 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.605927 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8cb7ae97-6473-478e-b8e9-534be9f60b02-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.606196 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.605990 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8cb7ae97-6473-478e-b8e9-534be9f60b02-root\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.606429 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.606205 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8cb7ae97-6473-478e-b8e9-534be9f60b02-node-exporter-textfile\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.606429 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.606343 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8cb7ae97-6473-478e-b8e9-534be9f60b02-node-exporter-accelerators-collector-config\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.606429 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.606392 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cb7ae97-6473-478e-b8e9-534be9f60b02-metrics-client-ca\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.608060 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.608041 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8cb7ae97-6473-478e-b8e9-534be9f60b02-node-exporter-tls\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.608144 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.608132 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8cb7ae97-6473-478e-b8e9-534be9f60b02-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.614028 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.614004 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pxvmp" Sep 05 06:28:10.615994 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.615975 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsxjb\" (UniqueName: \"kubernetes.io/projected/8cb7ae97-6473-478e-b8e9-534be9f60b02-kube-api-access-vsxjb\") pod \"node-exporter-v57dr\" (UID: \"8cb7ae97-6473-478e-b8e9-534be9f60b02\") " pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.632409 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.632379 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb" Sep 05 06:28:10.641621 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.641600 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v57dr" Sep 05 06:28:10.652275 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:28:10.652250 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb7ae97_6473_478e_b8e9_534be9f60b02.slice/crio-81da64debcafde4b41ea3389a094503f24231cd927c54e22e92ed92aaa45d2f8 WatchSource:0}: Error finding container 81da64debcafde4b41ea3389a094503f24231cd927c54e22e92ed92aaa45d2f8: Status 404 returned error can't find the container with id 81da64debcafde4b41ea3389a094503f24231cd927c54e22e92ed92aaa45d2f8 Sep 05 06:28:10.721184 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.721156 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v57dr" event={"ID":"8cb7ae97-6473-478e-b8e9-534be9f60b02","Type":"ContainerStarted","Data":"81da64debcafde4b41ea3389a094503f24231cd927c54e22e92ed92aaa45d2f8"} Sep 05 06:28:10.896084 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.896012 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-pxvmp"] Sep 05 06:28:10.899509 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:28:10.899480 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b45c53_3ea5_4817_b164_413cf5f2ae5d.slice/crio-d112aed6e0770f91a9234f4b2db094e84e9b64948576632cbb2c601df4bf8f70 WatchSource:0}: Error finding container d112aed6e0770f91a9234f4b2db094e84e9b64948576632cbb2c601df4bf8f70: Status 404 returned error can't find the container with id d112aed6e0770f91a9234f4b2db094e84e9b64948576632cbb2c601df4bf8f70 Sep 05 06:28:10.919905 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:10.919873 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb"] Sep 05 06:28:10.927210 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:28:10.927187 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac12b19_6ff2_41f9_a7d7_e4416e7962a6.slice/crio-a8ff371e2d8db0e073389ea11c67fbe520ce0fe40df53e3d6037a3fe4a04f64c WatchSource:0}: Error finding container a8ff371e2d8db0e073389ea11c67fbe520ce0fe40df53e3d6037a3fe4a04f64c: Status 404 returned error can't find the container with id a8ff371e2d8db0e073389ea11c67fbe520ce0fe40df53e3d6037a3fe4a04f64c Sep 05 06:28:11.727710 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:11.727676 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v57dr" event={"ID":"8cb7ae97-6473-478e-b8e9-534be9f60b02","Type":"ContainerStarted","Data":"9be9decf50d64eb184e4a3fbc1b3191a60baa449702fe2a7f6048b26e9f587dc"} Sep 05 06:28:11.729275 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:11.729241 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb" event={"ID":"5ac12b19-6ff2-41f9-a7d7-e4416e7962a6","Type":"ContainerStarted","Data":"a8ff371e2d8db0e073389ea11c67fbe520ce0fe40df53e3d6037a3fe4a04f64c"} Sep 05 06:28:11.734648 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:11.734617 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pxvmp" event={"ID":"03b45c53-3ea5-4817-b164-413cf5f2ae5d","Type":"ContainerStarted","Data":"5e44a13622e448a9c343a471876188bef41f8f3f5f26a42e9e4aa323a92f6280"} Sep 05 06:28:11.734761 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:11.734679 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pxvmp" event={"ID":"03b45c53-3ea5-4817-b164-413cf5f2ae5d","Type":"ContainerStarted","Data":"7dd6c816c246a9e398cc85ec5bf0b17864906c9c270f3070ab7a999843106a56"} Sep 05 06:28:11.734761 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:11.734709 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pxvmp" event={"ID":"03b45c53-3ea5-4817-b164-413cf5f2ae5d","Type":"ContainerStarted","Data":"d112aed6e0770f91a9234f4b2db094e84e9b64948576632cbb2c601df4bf8f70"} Sep 05 06:28:12.410479 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.410453 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn"] Sep 05 06:28:12.413691 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.413669 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.416485 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.416446 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 05 06:28:12.416485 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.416483 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-p2v5q\"" Sep 05 06:28:12.416674 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.416451 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 05 06:28:12.416674 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.416559 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 05 06:28:12.416884 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.416856 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 05 06:28:12.416963 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.416920 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 05 06:28:12.416963 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.416958 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4memgcccjt32k\"" Sep 05 06:28:12.426767 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.426749 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn"] Sep 05 06:28:12.523935 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.523659 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9590c37f-3669-4237-8ac8-78b14e17ce5b-secret-grpc-tls\") pod \"thanos-querier-7dcb5fb596-rsdbn\" (UID: \"9590c37f-3669-4237-8ac8-78b14e17ce5b\") " pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.523935 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.523695 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9590c37f-3669-4237-8ac8-78b14e17ce5b-secret-thanos-querier-tls\") pod \"thanos-querier-7dcb5fb596-rsdbn\" (UID: \"9590c37f-3669-4237-8ac8-78b14e17ce5b\") " pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.523935 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.523718 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9590c37f-3669-4237-8ac8-78b14e17ce5b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dcb5fb596-rsdbn\" (UID: \"9590c37f-3669-4237-8ac8-78b14e17ce5b\") " pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.523935 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.523745 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9590c37f-3669-4237-8ac8-78b14e17ce5b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dcb5fb596-rsdbn\" (UID: \"9590c37f-3669-4237-8ac8-78b14e17ce5b\") " pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.523935 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.523766 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2gxc\" (UniqueName: \"kubernetes.io/projected/9590c37f-3669-4237-8ac8-78b14e17ce5b-kube-api-access-z2gxc\") pod \"thanos-querier-7dcb5fb596-rsdbn\" (UID: \"9590c37f-3669-4237-8ac8-78b14e17ce5b\") " pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.523935 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.523787 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9590c37f-3669-4237-8ac8-78b14e17ce5b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dcb5fb596-rsdbn\" (UID: \"9590c37f-3669-4237-8ac8-78b14e17ce5b\") " pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.523935 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.523811 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9590c37f-3669-4237-8ac8-78b14e17ce5b-metrics-client-ca\") pod \"thanos-querier-7dcb5fb596-rsdbn\" (UID: \"9590c37f-3669-4237-8ac8-78b14e17ce5b\") " pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.523935 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.523831 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9590c37f-3669-4237-8ac8-78b14e17ce5b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dcb5fb596-rsdbn\" (UID: \"9590c37f-3669-4237-8ac8-78b14e17ce5b\") " pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.624857 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.624820 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9590c37f-3669-4237-8ac8-78b14e17ce5b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dcb5fb596-rsdbn\" (UID: \"9590c37f-3669-4237-8ac8-78b14e17ce5b\") " pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.624857 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.624858 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2gxc\" (UniqueName: \"kubernetes.io/projected/9590c37f-3669-4237-8ac8-78b14e17ce5b-kube-api-access-z2gxc\") pod \"thanos-querier-7dcb5fb596-rsdbn\" (UID: \"9590c37f-3669-4237-8ac8-78b14e17ce5b\") " pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.625092 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.624893 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9590c37f-3669-4237-8ac8-78b14e17ce5b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dcb5fb596-rsdbn\" (UID: \"9590c37f-3669-4237-8ac8-78b14e17ce5b\") " pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.625092 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.624933 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9590c37f-3669-4237-8ac8-78b14e17ce5b-metrics-client-ca\") pod \"thanos-querier-7dcb5fb596-rsdbn\" (UID: \"9590c37f-3669-4237-8ac8-78b14e17ce5b\") " pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.625092 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.624966 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9590c37f-3669-4237-8ac8-78b14e17ce5b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dcb5fb596-rsdbn\" (UID: \"9590c37f-3669-4237-8ac8-78b14e17ce5b\") " pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.625092 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.625047 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9590c37f-3669-4237-8ac8-78b14e17ce5b-secret-grpc-tls\") pod \"thanos-querier-7dcb5fb596-rsdbn\" (UID: \"9590c37f-3669-4237-8ac8-78b14e17ce5b\") " pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.625092 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.625076 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9590c37f-3669-4237-8ac8-78b14e17ce5b-secret-thanos-querier-tls\") pod \"thanos-querier-7dcb5fb596-rsdbn\" (UID: \"9590c37f-3669-4237-8ac8-78b14e17ce5b\") " pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.625353 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.625106 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9590c37f-3669-4237-8ac8-78b14e17ce5b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dcb5fb596-rsdbn\" (UID: \"9590c37f-3669-4237-8ac8-78b14e17ce5b\") " pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.625899 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.625825 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9590c37f-3669-4237-8ac8-78b14e17ce5b-metrics-client-ca\") pod \"thanos-querier-7dcb5fb596-rsdbn\" (UID: \"9590c37f-3669-4237-8ac8-78b14e17ce5b\") " pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.628291 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.628263 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9590c37f-3669-4237-8ac8-78b14e17ce5b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dcb5fb596-rsdbn\" (UID: \"9590c37f-3669-4237-8ac8-78b14e17ce5b\") " pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.628428 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.628263 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9590c37f-3669-4237-8ac8-78b14e17ce5b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dcb5fb596-rsdbn\" (UID: \"9590c37f-3669-4237-8ac8-78b14e17ce5b\") " pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.628428 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.628405 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9590c37f-3669-4237-8ac8-78b14e17ce5b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dcb5fb596-rsdbn\" (UID: \"9590c37f-3669-4237-8ac8-78b14e17ce5b\") " pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.628551 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.628521 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9590c37f-3669-4237-8ac8-78b14e17ce5b-secret-grpc-tls\") pod \"thanos-querier-7dcb5fb596-rsdbn\" (UID: \"9590c37f-3669-4237-8ac8-78b14e17ce5b\") " pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.628658 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.628633 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9590c37f-3669-4237-8ac8-78b14e17ce5b-secret-thanos-querier-tls\") pod \"thanos-querier-7dcb5fb596-rsdbn\" (UID: \"9590c37f-3669-4237-8ac8-78b14e17ce5b\") " pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.628885 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.628867 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9590c37f-3669-4237-8ac8-78b14e17ce5b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dcb5fb596-rsdbn\" (UID: \"9590c37f-3669-4237-8ac8-78b14e17ce5b\") " pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.634662 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.634640 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2gxc\" (UniqueName: \"kubernetes.io/projected/9590c37f-3669-4237-8ac8-78b14e17ce5b-kube-api-access-z2gxc\") pod \"thanos-querier-7dcb5fb596-rsdbn\" (UID: \"9590c37f-3669-4237-8ac8-78b14e17ce5b\") " pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.724506 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.724460 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:12.741015 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.740982 2565 generic.go:356] "Generic (PLEG): container finished" podID="8cb7ae97-6473-478e-b8e9-534be9f60b02" containerID="9be9decf50d64eb184e4a3fbc1b3191a60baa449702fe2a7f6048b26e9f587dc" exitCode=0 Sep 05 06:28:12.741354 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.741067 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v57dr" event={"ID":"8cb7ae97-6473-478e-b8e9-534be9f60b02","Type":"ContainerDied","Data":"9be9decf50d64eb184e4a3fbc1b3191a60baa449702fe2a7f6048b26e9f587dc"} Sep 05 06:28:12.744597 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.744576 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb" event={"ID":"5ac12b19-6ff2-41f9-a7d7-e4416e7962a6","Type":"ContainerStarted","Data":"01db4060e03b5244864a15303f8968cde612edfe6fa1cc93a7f4998e59832361"} Sep 05 06:28:12.744716 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.744632 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb" event={"ID":"5ac12b19-6ff2-41f9-a7d7-e4416e7962a6","Type":"ContainerStarted","Data":"2794fb391a8faf1f59dc06b2094a3e5189006ddf4d1af11791b2aed0fe7c34fe"} Sep 05 06:28:12.744716 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.744668 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb" event={"ID":"5ac12b19-6ff2-41f9-a7d7-e4416e7962a6","Type":"ContainerStarted","Data":"0d739f149bd46007fbeb8756fb803ceb21567eb8f5d34ca9b197b456e5755afa"} Sep 05 06:28:12.747115 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.747090 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pxvmp" event={"ID":"03b45c53-3ea5-4817-b164-413cf5f2ae5d","Type":"ContainerStarted","Data":"4a95d33f4c7061406093a596bc590511d314ee27fcdd41acf442d6498d720b7f"} Sep 05 06:28:12.786516 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.785866 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pxvmp" podStartSLOduration=1.426894336 podStartE2EDuration="2.78584544s" totalImagesPullingTime="1.358951104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:28:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:28:12.784761531 +0000 UTC m=+117.169304649" watchObservedRunningTime="2026-09-05 06:28:12.78584544 +0000 UTC m=+117.170388565" Sep 05 06:28:12.806756 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:12.806703 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-zwhpb" podStartSLOduration=1.375842101 podStartE2EDuration="2.806682233s" totalImagesPullingTime="1.430840132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:28:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:28:12.805911954 +0000 UTC m=+117.190455081" watchObservedRunningTime="2026-09-05 06:28:12.806682233 +0000 UTC m=+117.191225357" Sep 05 06:28:13.007170 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:13.007102 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn"] Sep 05 06:28:13.011426 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:28:13.011399 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9590c37f_3669_4237_8ac8_78b14e17ce5b.slice/crio-153f8674d14eb72e0bbb0b68c9cfedbf5089c2df98977a533b05512b35fb2e54 WatchSource:0}: Error finding container 153f8674d14eb72e0bbb0b68c9cfedbf5089c2df98977a533b05512b35fb2e54: Status 404 returned error can't find the container with id 153f8674d14eb72e0bbb0b68c9cfedbf5089c2df98977a533b05512b35fb2e54 Sep 05 06:28:13.753554 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:13.753519 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v57dr" event={"ID":"8cb7ae97-6473-478e-b8e9-534be9f60b02","Type":"ContainerStarted","Data":"6c2fa579a141af694574ecd2f9642333e2ce5c0642c099caf1a062960a0fc60b"} Sep 05 06:28:13.754025 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:13.753572 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v57dr" event={"ID":"8cb7ae97-6473-478e-b8e9-534be9f60b02","Type":"ContainerStarted","Data":"629b02e7fd123654878b895233842a62549614daf40854b2258eaae386c3175a"} Sep 05 06:28:13.754660 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:13.754635 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" event={"ID":"9590c37f-3669-4237-8ac8-78b14e17ce5b","Type":"ContainerStarted","Data":"153f8674d14eb72e0bbb0b68c9cfedbf5089c2df98977a533b05512b35fb2e54"} Sep 05 06:28:13.776995 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:13.776934 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-v57dr" podStartSLOduration=2.712410566 podStartE2EDuration="3.77691439s" totalImagesPullingTime="981.966649ms" totalInitContainerRuntime="82.537175ms" isStatefulPod=true podCreationTimestamp="2026-09-05 06:28:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:28:13.775945494 +0000 UTC m=+118.160488639" watchObservedRunningTime="2026-09-05 06:28:13.77691439 +0000 UTC m=+118.161457737" Sep 05 06:28:14.716008 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.715960 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7c4df55d4f-tw9lb"] Sep 05 06:28:14.718758 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.718731 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:14.721439 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.721415 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-wks6r\"" Sep 05 06:28:14.721586 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.721537 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 05 06:28:14.721586 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.721561 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 05 06:28:14.722180 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.722162 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 05 06:28:14.722495 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.722467 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 05 06:28:14.722600 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.722555 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-327qckun7nm6p\"" Sep 05 06:28:14.729863 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.729842 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7c4df55d4f-tw9lb"] Sep 05 06:28:14.844497 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.844447 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bceafc69-35f3-4e13-9b38-0024cbfa8899-client-ca-bundle\") pod \"metrics-server-7c4df55d4f-tw9lb\" (UID: \"bceafc69-35f3-4e13-9b38-0024cbfa8899\") " pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:14.844497 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.844494 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bceafc69-35f3-4e13-9b38-0024cbfa8899-secret-metrics-server-tls\") pod \"metrics-server-7c4df55d4f-tw9lb\" (UID: \"bceafc69-35f3-4e13-9b38-0024cbfa8899\") " pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:14.844983 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.844529 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bceafc69-35f3-4e13-9b38-0024cbfa8899-metrics-server-audit-profiles\") pod \"metrics-server-7c4df55d4f-tw9lb\" (UID: \"bceafc69-35f3-4e13-9b38-0024cbfa8899\") " pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:14.844983 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.844639 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bceafc69-35f3-4e13-9b38-0024cbfa8899-audit-log\") pod \"metrics-server-7c4df55d4f-tw9lb\" (UID: \"bceafc69-35f3-4e13-9b38-0024cbfa8899\") " pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:14.844983 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.844691 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bceafc69-35f3-4e13-9b38-0024cbfa8899-secret-metrics-server-client-certs\") pod \"metrics-server-7c4df55d4f-tw9lb\" (UID: \"bceafc69-35f3-4e13-9b38-0024cbfa8899\") " pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:14.844983 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.844752 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcr4m\" (UniqueName: \"kubernetes.io/projected/bceafc69-35f3-4e13-9b38-0024cbfa8899-kube-api-access-pcr4m\") pod \"metrics-server-7c4df55d4f-tw9lb\" (UID: \"bceafc69-35f3-4e13-9b38-0024cbfa8899\") " pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:14.844983 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.844845 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bceafc69-35f3-4e13-9b38-0024cbfa8899-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c4df55d4f-tw9lb\" (UID: \"bceafc69-35f3-4e13-9b38-0024cbfa8899\") " pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:14.945928 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.945907 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bceafc69-35f3-4e13-9b38-0024cbfa8899-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c4df55d4f-tw9lb\" (UID: \"bceafc69-35f3-4e13-9b38-0024cbfa8899\") " pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:14.946032 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.945958 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bceafc69-35f3-4e13-9b38-0024cbfa8899-client-ca-bundle\") pod \"metrics-server-7c4df55d4f-tw9lb\" (UID: \"bceafc69-35f3-4e13-9b38-0024cbfa8899\") " pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:14.946032 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.945986 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bceafc69-35f3-4e13-9b38-0024cbfa8899-secret-metrics-server-tls\") pod \"metrics-server-7c4df55d4f-tw9lb\" (UID: \"bceafc69-35f3-4e13-9b38-0024cbfa8899\") " pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:14.946032 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.946017 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bceafc69-35f3-4e13-9b38-0024cbfa8899-metrics-server-audit-profiles\") pod \"metrics-server-7c4df55d4f-tw9lb\" (UID: \"bceafc69-35f3-4e13-9b38-0024cbfa8899\") " pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:14.946189 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.946084 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bceafc69-35f3-4e13-9b38-0024cbfa8899-audit-log\") pod \"metrics-server-7c4df55d4f-tw9lb\" (UID: \"bceafc69-35f3-4e13-9b38-0024cbfa8899\") " pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:14.946189 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.946133 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bceafc69-35f3-4e13-9b38-0024cbfa8899-secret-metrics-server-client-certs\") pod \"metrics-server-7c4df55d4f-tw9lb\" (UID: \"bceafc69-35f3-4e13-9b38-0024cbfa8899\") " pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:14.946189 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.946177 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcr4m\" (UniqueName: \"kubernetes.io/projected/bceafc69-35f3-4e13-9b38-0024cbfa8899-kube-api-access-pcr4m\") pod \"metrics-server-7c4df55d4f-tw9lb\" (UID: \"bceafc69-35f3-4e13-9b38-0024cbfa8899\") " pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:14.946760 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.946706 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bceafc69-35f3-4e13-9b38-0024cbfa8899-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c4df55d4f-tw9lb\" (UID: \"bceafc69-35f3-4e13-9b38-0024cbfa8899\") " pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:14.946899 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.946795 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bceafc69-35f3-4e13-9b38-0024cbfa8899-audit-log\") pod \"metrics-server-7c4df55d4f-tw9lb\" (UID: \"bceafc69-35f3-4e13-9b38-0024cbfa8899\") " pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:14.947291 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.947269 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bceafc69-35f3-4e13-9b38-0024cbfa8899-metrics-server-audit-profiles\") pod \"metrics-server-7c4df55d4f-tw9lb\" (UID: \"bceafc69-35f3-4e13-9b38-0024cbfa8899\") " pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:14.948823 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.948800 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bceafc69-35f3-4e13-9b38-0024cbfa8899-secret-metrics-server-tls\") pod \"metrics-server-7c4df55d4f-tw9lb\" (UID: \"bceafc69-35f3-4e13-9b38-0024cbfa8899\") " pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:14.948915 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.948899 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bceafc69-35f3-4e13-9b38-0024cbfa8899-client-ca-bundle\") pod \"metrics-server-7c4df55d4f-tw9lb\" (UID: \"bceafc69-35f3-4e13-9b38-0024cbfa8899\") " pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:14.948957 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.948937 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bceafc69-35f3-4e13-9b38-0024cbfa8899-secret-metrics-server-client-certs\") pod \"metrics-server-7c4df55d4f-tw9lb\" (UID: \"bceafc69-35f3-4e13-9b38-0024cbfa8899\") " pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:14.956480 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:14.956461 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcr4m\" (UniqueName: \"kubernetes.io/projected/bceafc69-35f3-4e13-9b38-0024cbfa8899-kube-api-access-pcr4m\") pod \"metrics-server-7c4df55d4f-tw9lb\" (UID: \"bceafc69-35f3-4e13-9b38-0024cbfa8899\") " pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:15.030607 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:15.030580 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:15.108804 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:15.108774 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-tlf9c"] Sep 05 06:28:15.111858 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:15.111836 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tlf9c" Sep 05 06:28:15.115944 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:15.115465 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5l97n\"" Sep 05 06:28:15.115944 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:15.115668 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 05 06:28:15.126706 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:15.126677 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-tlf9c"] Sep 05 06:28:15.248242 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:15.248158 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkqpb\" (UniqueName: \"kubernetes.io/projected/0d6de61d-1d35-4d43-b766-a3419dc05d81-kube-api-access-mkqpb\") pod \"monitoring-plugin-85bbbbbf4-tlf9c\" (UID: \"0d6de61d-1d35-4d43-b766-a3419dc05d81\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tlf9c" Sep 05 06:28:15.248408 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:15.248280 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d6de61d-1d35-4d43-b766-a3419dc05d81-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-tlf9c\" (UID: \"0d6de61d-1d35-4d43-b766-a3419dc05d81\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tlf9c" Sep 05 06:28:15.308135 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:15.308111 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7c4df55d4f-tw9lb"] Sep 05 06:28:15.312767 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:28:15.312742 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbceafc69_35f3_4e13_9b38_0024cbfa8899.slice/crio-542340ba39cd388dbf13e8cca01f9c5cd404db4df14d5c823c2d496cf30bf792 WatchSource:0}: Error finding container 542340ba39cd388dbf13e8cca01f9c5cd404db4df14d5c823c2d496cf30bf792: Status 404 returned error can't find the container with id 542340ba39cd388dbf13e8cca01f9c5cd404db4df14d5c823c2d496cf30bf792 Sep 05 06:28:15.349384 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:15.349357 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d6de61d-1d35-4d43-b766-a3419dc05d81-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-tlf9c\" (UID: \"0d6de61d-1d35-4d43-b766-a3419dc05d81\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tlf9c" Sep 05 06:28:15.349499 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:15.349398 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkqpb\" (UniqueName: \"kubernetes.io/projected/0d6de61d-1d35-4d43-b766-a3419dc05d81-kube-api-access-mkqpb\") pod \"monitoring-plugin-85bbbbbf4-tlf9c\" (UID: \"0d6de61d-1d35-4d43-b766-a3419dc05d81\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tlf9c" Sep 05 06:28:15.349535 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:28:15.349502 2565 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 05 06:28:15.349572 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:28:15.349565 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d6de61d-1d35-4d43-b766-a3419dc05d81-monitoring-plugin-cert podName:0d6de61d-1d35-4d43-b766-a3419dc05d81 nodeName:}" failed. No retries permitted until 2026-09-05 06:28:15.849549003 +0000 UTC m=+120.234092121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/0d6de61d-1d35-4d43-b766-a3419dc05d81-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-tlf9c" (UID: "0d6de61d-1d35-4d43-b766-a3419dc05d81") : secret "monitoring-plugin-cert" not found Sep 05 06:28:15.359257 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:15.359229 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkqpb\" (UniqueName: \"kubernetes.io/projected/0d6de61d-1d35-4d43-b766-a3419dc05d81-kube-api-access-mkqpb\") pod \"monitoring-plugin-85bbbbbf4-tlf9c\" (UID: \"0d6de61d-1d35-4d43-b766-a3419dc05d81\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tlf9c" Sep 05 06:28:15.764461 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:15.764392 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" event={"ID":"bceafc69-35f3-4e13-9b38-0024cbfa8899","Type":"ContainerStarted","Data":"542340ba39cd388dbf13e8cca01f9c5cd404db4df14d5c823c2d496cf30bf792"} Sep 05 06:28:15.766814 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:15.766782 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" event={"ID":"9590c37f-3669-4237-8ac8-78b14e17ce5b","Type":"ContainerStarted","Data":"ca896b9ca4a41c98ba73aa0a297f23cdba77202a6832678f2d48f6736cd77220"} Sep 05 06:28:15.766946 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:15.766851 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" event={"ID":"9590c37f-3669-4237-8ac8-78b14e17ce5b","Type":"ContainerStarted","Data":"b43dcc599281ae347970f36715f2f6ba50910b1abbe44d951cb025723c2e3df5"} Sep 05 06:28:15.766946 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:15.766880 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" event={"ID":"9590c37f-3669-4237-8ac8-78b14e17ce5b","Type":"ContainerStarted","Data":"6d808d1d73f12dba114e14c62931af0cc57bde7d62f2fb63fca8637262128be5"} Sep 05 06:28:15.854717 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:15.854694 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d6de61d-1d35-4d43-b766-a3419dc05d81-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-tlf9c\" (UID: \"0d6de61d-1d35-4d43-b766-a3419dc05d81\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tlf9c" Sep 05 06:28:15.857663 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:15.857640 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d6de61d-1d35-4d43-b766-a3419dc05d81-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-tlf9c\" (UID: \"0d6de61d-1d35-4d43-b766-a3419dc05d81\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tlf9c" Sep 05 06:28:16.037669 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:16.037643 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5l97n\"" Sep 05 06:28:16.045051 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:16.045028 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tlf9c" Sep 05 06:28:16.337086 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:16.334487 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-tlf9c"] Sep 05 06:28:16.609708 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:28:16.609675 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6de61d_1d35_4d43_b766_a3419dc05d81.slice/crio-f3477879e616bb0fc370158b9160dd563cecd66eddc0e7471fad435f7d139ddd WatchSource:0}: Error finding container f3477879e616bb0fc370158b9160dd563cecd66eddc0e7471fad435f7d139ddd: Status 404 returned error can't find the container with id f3477879e616bb0fc370158b9160dd563cecd66eddc0e7471fad435f7d139ddd Sep 05 06:28:16.772133 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:16.772096 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" event={"ID":"bceafc69-35f3-4e13-9b38-0024cbfa8899","Type":"ContainerStarted","Data":"601628d233df5e665b0759ac1e054663d0937c0bf502552a05b32b77705c944b"} Sep 05 06:28:16.773356 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:16.773333 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tlf9c" event={"ID":"0d6de61d-1d35-4d43-b766-a3419dc05d81","Type":"ContainerStarted","Data":"f3477879e616bb0fc370158b9160dd563cecd66eddc0e7471fad435f7d139ddd"} Sep 05 06:28:16.776348 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:16.776301 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" event={"ID":"9590c37f-3669-4237-8ac8-78b14e17ce5b","Type":"ContainerStarted","Data":"534f14163d18235ab7fbffb7fe0fc1e609cabd458601a315561663dbaca291f1"} Sep 05 06:28:16.776529 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:16.776388 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" event={"ID":"9590c37f-3669-4237-8ac8-78b14e17ce5b","Type":"ContainerStarted","Data":"ec368ad819f7008d726ab6d5549c603c879c5881e4135116b072e15528bf347b"} Sep 05 06:28:16.776529 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:16.776435 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" event={"ID":"9590c37f-3669-4237-8ac8-78b14e17ce5b","Type":"ContainerStarted","Data":"385f8da92ec024e5f64f417d2c01eae64b3b6baffc6b13e56a1dbbd21bd36c41"} Sep 05 06:28:16.776529 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:16.776495 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:16.792346 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:16.792273 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" podStartSLOduration=1.46512888 podStartE2EDuration="2.792259828s" totalImagesPullingTime="1.327130948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:28:16.79079047 +0000 UTC m=+121.175333619" watchObservedRunningTime="2026-09-05 06:28:16.792259828 +0000 UTC m=+121.176802946" Sep 05 06:28:16.812972 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:16.812931 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" podStartSLOduration=2.191229329 podStartE2EDuration="4.812917118s" totalImagesPullingTime="2.621687789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:28:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:28:16.811878816 +0000 UTC m=+121.196421941" watchObservedRunningTime="2026-09-05 06:28:16.812917118 +0000 UTC m=+121.197460243" Sep 05 06:28:16.985924 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:16.985776 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:28:18.785127 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:18.785095 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tlf9c" event={"ID":"0d6de61d-1d35-4d43-b766-a3419dc05d81","Type":"ContainerStarted","Data":"25e40bd9841acdc3b7cc2424011e59f7de45289a86439320fbf0efb2a9803cf3"} Sep 05 06:28:18.785576 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:18.785234 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tlf9c" Sep 05 06:28:18.790259 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:18.790229 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tlf9c" Sep 05 06:28:18.802918 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:18.802875 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tlf9c" podStartSLOduration=2.632322538 podStartE2EDuration="3.802861946s" totalImagesPullingTime="1.170539408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:28:18.801452411 +0000 UTC m=+123.185995537" watchObservedRunningTime="2026-09-05 06:28:18.802861946 +0000 UTC m=+123.187405070" Sep 05 06:28:20.677160 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:20.677127 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7759f7f78b-bvxqx" Sep 05 06:28:22.000274 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.000233 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" podUID="6653acf2-739a-4628-8f9f-e0658b4a9338" containerName="registry" containerID="cri-o://a87b9f012d6543f9c1e5e2657aaf5261b8928dc876c1be3f28e8e648b9f7a9c3" gracePeriod=30 Sep 05 06:28:22.244139 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.244113 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:28:22.313806 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.313727 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-tls\") pod \"6653acf2-739a-4628-8f9f-e0658b4a9338\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " Sep 05 06:28:22.313806 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.313800 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-certificates\") pod \"6653acf2-739a-4628-8f9f-e0658b4a9338\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " Sep 05 06:28:22.313980 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.313826 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6653acf2-739a-4628-8f9f-e0658b4a9338-image-registry-private-configuration\") pod \"6653acf2-739a-4628-8f9f-e0658b4a9338\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " Sep 05 06:28:22.313980 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.313854 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6653acf2-739a-4628-8f9f-e0658b4a9338-ca-trust-extracted\") pod \"6653acf2-739a-4628-8f9f-e0658b4a9338\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " Sep 05 06:28:22.313980 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.313887 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-bound-sa-token\") pod \"6653acf2-739a-4628-8f9f-e0658b4a9338\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " Sep 05 06:28:22.313980 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.313927 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6653acf2-739a-4628-8f9f-e0658b4a9338-trusted-ca\") pod \"6653acf2-739a-4628-8f9f-e0658b4a9338\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " Sep 05 06:28:22.313980 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.313950 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f99bb\" (UniqueName: \"kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-kube-api-access-f99bb\") pod \"6653acf2-739a-4628-8f9f-e0658b4a9338\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " Sep 05 06:28:22.314200 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.313999 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6653acf2-739a-4628-8f9f-e0658b4a9338-installation-pull-secrets\") pod \"6653acf2-739a-4628-8f9f-e0658b4a9338\" (UID: \"6653acf2-739a-4628-8f9f-e0658b4a9338\") " Sep 05 06:28:22.314288 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.314259 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6653acf2-739a-4628-8f9f-e0658b4a9338" (UID: "6653acf2-739a-4628-8f9f-e0658b4a9338"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:28:22.314916 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.314834 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6653acf2-739a-4628-8f9f-e0658b4a9338-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6653acf2-739a-4628-8f9f-e0658b4a9338" (UID: "6653acf2-739a-4628-8f9f-e0658b4a9338"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:28:22.316797 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.316770 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6653acf2-739a-4628-8f9f-e0658b4a9338" (UID: "6653acf2-739a-4628-8f9f-e0658b4a9338"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:28:22.316912 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.316799 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-kube-api-access-f99bb" (OuterVolumeSpecName: "kube-api-access-f99bb") pod "6653acf2-739a-4628-8f9f-e0658b4a9338" (UID: "6653acf2-739a-4628-8f9f-e0658b4a9338"). InnerVolumeSpecName "kube-api-access-f99bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:28:22.316912 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.316818 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6653acf2-739a-4628-8f9f-e0658b4a9338-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6653acf2-739a-4628-8f9f-e0658b4a9338" (UID: "6653acf2-739a-4628-8f9f-e0658b4a9338"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 06:28:22.317021 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.316923 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6653acf2-739a-4628-8f9f-e0658b4a9338-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6653acf2-739a-4628-8f9f-e0658b4a9338" (UID: "6653acf2-739a-4628-8f9f-e0658b4a9338"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 06:28:22.317021 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.316925 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6653acf2-739a-4628-8f9f-e0658b4a9338" (UID: "6653acf2-739a-4628-8f9f-e0658b4a9338"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:28:22.325132 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.325109 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6653acf2-739a-4628-8f9f-e0658b4a9338-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6653acf2-739a-4628-8f9f-e0658b4a9338" (UID: "6653acf2-739a-4628-8f9f-e0658b4a9338"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:28:22.415558 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.415526 2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6653acf2-739a-4628-8f9f-e0658b4a9338-ca-trust-extracted\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:28:22.415558 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.415553 2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-bound-sa-token\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:28:22.415558 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.415564 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6653acf2-739a-4628-8f9f-e0658b4a9338-trusted-ca\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:28:22.415764 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.415572 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f99bb\" (UniqueName: \"kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-kube-api-access-f99bb\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:28:22.415764 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.415581 2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6653acf2-739a-4628-8f9f-e0658b4a9338-installation-pull-secrets\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:28:22.415764 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.415589 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-tls\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:28:22.415764 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.415597 2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6653acf2-739a-4628-8f9f-e0658b4a9338-registry-certificates\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:28:22.415764 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.415606 2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6653acf2-739a-4628-8f9f-e0658b4a9338-image-registry-private-configuration\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:28:22.786444 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.786418 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7dcb5fb596-rsdbn" Sep 05 06:28:22.801076 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.801047 2565 generic.go:356] "Generic (PLEG): container finished" podID="6653acf2-739a-4628-8f9f-e0658b4a9338" containerID="a87b9f012d6543f9c1e5e2657aaf5261b8928dc876c1be3f28e8e648b9f7a9c3" exitCode=0 Sep 05 06:28:22.801205 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.801091 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" event={"ID":"6653acf2-739a-4628-8f9f-e0658b4a9338","Type":"ContainerDied","Data":"a87b9f012d6543f9c1e5e2657aaf5261b8928dc876c1be3f28e8e648b9f7a9c3"} Sep 05 06:28:22.801205 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.801095 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" Sep 05 06:28:22.801205 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.801137 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bd7fcc894-s297m" event={"ID":"6653acf2-739a-4628-8f9f-e0658b4a9338","Type":"ContainerDied","Data":"9dbc62b5f50374e3181f9b9a975223d75d7cf559a0141e2f525b0d7061f104e6"} Sep 05 06:28:22.801205 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.801158 2565 scope.go:117] "RemoveContainer" containerID="a87b9f012d6543f9c1e5e2657aaf5261b8928dc876c1be3f28e8e648b9f7a9c3" Sep 05 06:28:22.809784 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.809765 2565 scope.go:117] "RemoveContainer" containerID="a87b9f012d6543f9c1e5e2657aaf5261b8928dc876c1be3f28e8e648b9f7a9c3" Sep 05 06:28:22.810044 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:28:22.810023 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a87b9f012d6543f9c1e5e2657aaf5261b8928dc876c1be3f28e8e648b9f7a9c3\": container with ID starting with a87b9f012d6543f9c1e5e2657aaf5261b8928dc876c1be3f28e8e648b9f7a9c3 not found: ID does not exist" containerID="a87b9f012d6543f9c1e5e2657aaf5261b8928dc876c1be3f28e8e648b9f7a9c3" Sep 05 06:28:22.810112 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.810058 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a87b9f012d6543f9c1e5e2657aaf5261b8928dc876c1be3f28e8e648b9f7a9c3"} err="failed to get container status \"a87b9f012d6543f9c1e5e2657aaf5261b8928dc876c1be3f28e8e648b9f7a9c3\": rpc error: code = NotFound desc = could not find container \"a87b9f012d6543f9c1e5e2657aaf5261b8928dc876c1be3f28e8e648b9f7a9c3\": container with ID starting with a87b9f012d6543f9c1e5e2657aaf5261b8928dc876c1be3f28e8e648b9f7a9c3 not found: ID does not exist" Sep 05 06:28:22.831211 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.831186 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bd7fcc894-s297m"] Sep 05 06:28:22.836017 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:22.835993 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7bd7fcc894-s297m"] Sep 05 06:28:23.156228 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.156194 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5457d4cb6c-lv5kj"] Sep 05 06:28:23.156609 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.156544 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6653acf2-739a-4628-8f9f-e0658b4a9338" containerName="registry" Sep 05 06:28:23.156609 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.156556 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6653acf2-739a-4628-8f9f-e0658b4a9338" containerName="registry" Sep 05 06:28:23.156712 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.156636 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6653acf2-739a-4628-8f9f-e0658b4a9338" containerName="registry" Sep 05 06:28:23.161947 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.161928 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:28:23.164979 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.164957 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 05 06:28:23.165256 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.165238 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 05 06:28:23.165403 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.165379 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 05 06:28:23.165531 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.165386 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 05 06:28:23.165531 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.165411 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 05 06:28:23.165685 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.165667 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7r4f6\"" Sep 05 06:28:23.165743 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.165715 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 05 06:28:23.165803 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.165741 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 05 06:28:23.173404 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.173384 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5457d4cb6c-lv5kj"] Sep 05 06:28:23.220997 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.220960 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f53c51f-6fcb-4241-ad84-28d96684b3e6-console-config\") pod \"console-5457d4cb6c-lv5kj\" (UID: \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\") " pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:28:23.220997 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.221000 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f53c51f-6fcb-4241-ad84-28d96684b3e6-console-serving-cert\") pod \"console-5457d4cb6c-lv5kj\" (UID: \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\") " pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:28:23.221224 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.221102 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f53c51f-6fcb-4241-ad84-28d96684b3e6-console-oauth-config\") pod \"console-5457d4cb6c-lv5kj\" (UID: \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\") " pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:28:23.221224 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.221146 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f53c51f-6fcb-4241-ad84-28d96684b3e6-oauth-serving-cert\") pod \"console-5457d4cb6c-lv5kj\" (UID: \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\") " pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:28:23.221224 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.221164 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f53c51f-6fcb-4241-ad84-28d96684b3e6-service-ca\") pod \"console-5457d4cb6c-lv5kj\" (UID: \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\") " pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:28:23.221224 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.221215 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrjm5\" (UniqueName: \"kubernetes.io/projected/3f53c51f-6fcb-4241-ad84-28d96684b3e6-kube-api-access-vrjm5\") pod \"console-5457d4cb6c-lv5kj\" (UID: \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\") " pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:28:23.322385 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.322351 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrjm5\" (UniqueName: \"kubernetes.io/projected/3f53c51f-6fcb-4241-ad84-28d96684b3e6-kube-api-access-vrjm5\") pod \"console-5457d4cb6c-lv5kj\" (UID: \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\") " pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:28:23.322565 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.322417 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f53c51f-6fcb-4241-ad84-28d96684b3e6-console-config\") pod \"console-5457d4cb6c-lv5kj\" (UID: \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\") " pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:28:23.322565 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.322449 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f53c51f-6fcb-4241-ad84-28d96684b3e6-console-serving-cert\") pod \"console-5457d4cb6c-lv5kj\" (UID: \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\") " pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:28:23.322565 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.322530 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f53c51f-6fcb-4241-ad84-28d96684b3e6-console-oauth-config\") pod \"console-5457d4cb6c-lv5kj\" (UID: \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\") " pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:28:23.322742 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.322556 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f53c51f-6fcb-4241-ad84-28d96684b3e6-oauth-serving-cert\") pod \"console-5457d4cb6c-lv5kj\" (UID: \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\") " pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:28:23.322742 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.322607 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f53c51f-6fcb-4241-ad84-28d96684b3e6-service-ca\") pod \"console-5457d4cb6c-lv5kj\" (UID: \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\") " pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:28:23.323116 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.323087 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f53c51f-6fcb-4241-ad84-28d96684b3e6-console-config\") pod \"console-5457d4cb6c-lv5kj\" (UID: \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\") " pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:28:23.323414 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.323195 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f53c51f-6fcb-4241-ad84-28d96684b3e6-oauth-serving-cert\") pod \"console-5457d4cb6c-lv5kj\" (UID: \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\") " pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:28:23.323504 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.323482 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f53c51f-6fcb-4241-ad84-28d96684b3e6-service-ca\") pod \"console-5457d4cb6c-lv5kj\" (UID: \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\") " pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:28:23.325070 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.325049 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f53c51f-6fcb-4241-ad84-28d96684b3e6-console-oauth-config\") pod \"console-5457d4cb6c-lv5kj\" (UID: \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\") " pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:28:23.325193 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.325170 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f53c51f-6fcb-4241-ad84-28d96684b3e6-console-serving-cert\") pod \"console-5457d4cb6c-lv5kj\" (UID: \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\") " pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:28:23.332689 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.332662 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrjm5\" (UniqueName: \"kubernetes.io/projected/3f53c51f-6fcb-4241-ad84-28d96684b3e6-kube-api-access-vrjm5\") pod \"console-5457d4cb6c-lv5kj\" (UID: \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\") " pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:28:23.470818 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.470724 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:28:23.754367 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.754256 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5457d4cb6c-lv5kj"] Sep 05 06:28:23.759880 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:28:23.759844 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f53c51f_6fcb_4241_ad84_28d96684b3e6.slice/crio-f912789af287bc91768d2a06c5765c551aeaad5b08b0addcf61485f36d278c6a WatchSource:0}: Error finding container f912789af287bc91768d2a06c5765c551aeaad5b08b0addcf61485f36d278c6a: Status 404 returned error can't find the container with id f912789af287bc91768d2a06c5765c551aeaad5b08b0addcf61485f36d278c6a Sep 05 06:28:23.805827 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:23.805791 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5457d4cb6c-lv5kj" event={"ID":"3f53c51f-6fcb-4241-ad84-28d96684b3e6","Type":"ContainerStarted","Data":"f912789af287bc91768d2a06c5765c551aeaad5b08b0addcf61485f36d278c6a"} Sep 05 06:28:24.165900 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:24.165861 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6653acf2-739a-4628-8f9f-e0658b4a9338" path="/var/lib/kubelet/pods/6653acf2-739a-4628-8f9f-e0658b4a9338/volumes" Sep 05 06:28:25.946040 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:25.945999 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs\") pod \"network-metrics-daemon-545p9\" (UID: \"9103306e-ef91-4539-b914-8c3cdb23ea94\") " pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:28:25.948958 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:25.948922 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9103306e-ef91-4539-b914-8c3cdb23ea94-metrics-certs\") pod \"network-metrics-daemon-545p9\" (UID: \"9103306e-ef91-4539-b914-8c3cdb23ea94\") " pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:28:26.071370 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:26.071331 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4m5kh\"" Sep 05 06:28:26.079408 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:26.079379 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-545p9" Sep 05 06:28:26.488388 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:26.488290 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-545p9"] Sep 05 06:28:26.491494 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:28:26.491464 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9103306e_ef91_4539_b914_8c3cdb23ea94.slice/crio-ad0a2dd3b895ef3a42c35fbc2ad795dd42a018d3d4d38e7e47ff74c2ee1a76f2 WatchSource:0}: Error finding container ad0a2dd3b895ef3a42c35fbc2ad795dd42a018d3d4d38e7e47ff74c2ee1a76f2: Status 404 returned error can't find the container with id ad0a2dd3b895ef3a42c35fbc2ad795dd42a018d3d4d38e7e47ff74c2ee1a76f2 Sep 05 06:28:26.822142 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:26.822035 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5457d4cb6c-lv5kj" event={"ID":"3f53c51f-6fcb-4241-ad84-28d96684b3e6","Type":"ContainerStarted","Data":"72a3891886c8130ef81257dc4ef3838998fa9635643631f27cc1925f673339e4"} Sep 05 06:28:26.823289 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:26.823266 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-545p9" event={"ID":"9103306e-ef91-4539-b914-8c3cdb23ea94","Type":"ContainerStarted","Data":"ad0a2dd3b895ef3a42c35fbc2ad795dd42a018d3d4d38e7e47ff74c2ee1a76f2"} Sep 05 06:28:26.841814 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:26.841747 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5457d4cb6c-lv5kj" podStartSLOduration=1.330554211 podStartE2EDuration="3.841730005s" totalImagesPullingTime="2.511175794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:28:26.841017947 +0000 UTC m=+131.225561072" watchObservedRunningTime="2026-09-05 06:28:26.841730005 +0000 UTC m=+131.226273128" Sep 05 06:28:27.829097 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:27.829003 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-545p9" event={"ID":"9103306e-ef91-4539-b914-8c3cdb23ea94","Type":"ContainerStarted","Data":"86ca7736693d13915f0e58f72dea342771ec2fe103ed514e7b7b135a3494bd43"} Sep 05 06:28:27.829484 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:27.829099 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-545p9" event={"ID":"9103306e-ef91-4539-b914-8c3cdb23ea94","Type":"ContainerStarted","Data":"1fa94825e78337f66b4d195b65c38086b4b7e422ea910799d5c914647a9f2cec"} Sep 05 06:28:27.846786 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:27.846735 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-545p9" podStartSLOduration=130.890621609 podStartE2EDuration="2m11.846717851s" totalImagesPullingTime="956.096242ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:28:27.846689242 +0000 UTC m=+132.231232367" watchObservedRunningTime="2026-09-05 06:28:27.846717851 +0000 UTC m=+132.231260966" Sep 05 06:28:33.471024 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:33.470979 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:28:33.471615 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:33.471054 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:28:33.475896 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:33.475873 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:28:33.856638 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:33.856611 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:28:35.031403 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:35.031375 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:35.031773 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:35.031427 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:42.635279 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:42.635241 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5457d4cb6c-lv5kj"] Sep 05 06:28:46.905092 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:46.905057 2565 generic.go:356] "Generic (PLEG): container finished" podID="62636c45-1709-4102-bbfc-a13e5f699d1e" containerID="24d5d75b2496dc4c02b2a36ec4e4cc844ebaa9adf8a1d783cbecf18d167dd90b" exitCode=0 Sep 05 06:28:46.905611 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:46.905110 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k68n5" event={"ID":"62636c45-1709-4102-bbfc-a13e5f699d1e","Type":"ContainerDied","Data":"24d5d75b2496dc4c02b2a36ec4e4cc844ebaa9adf8a1d783cbecf18d167dd90b"} Sep 05 06:28:46.905611 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:46.905565 2565 scope.go:117] "RemoveContainer" containerID="24d5d75b2496dc4c02b2a36ec4e4cc844ebaa9adf8a1d783cbecf18d167dd90b" Sep 05 06:28:47.910602 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:47.910570 2565 generic.go:356] "Generic (PLEG): container finished" podID="01815101-b04b-43ea-bb6c-bc896fd4b969" containerID="43d3fd6287bae5db9310d22609a22ca71090b3d176292c92988f944a7bf8855e" exitCode=0 Sep 05 06:28:47.911022 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:47.910652 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-q8tzt" event={"ID":"01815101-b04b-43ea-bb6c-bc896fd4b969","Type":"ContainerDied","Data":"43d3fd6287bae5db9310d22609a22ca71090b3d176292c92988f944a7bf8855e"} Sep 05 06:28:47.911022 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:47.911016 2565 scope.go:117] "RemoveContainer" containerID="43d3fd6287bae5db9310d22609a22ca71090b3d176292c92988f944a7bf8855e" Sep 05 06:28:47.912673 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:47.912651 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k68n5" event={"ID":"62636c45-1709-4102-bbfc-a13e5f699d1e","Type":"ContainerStarted","Data":"fe4be474db27cff75afeadf6106a340d5bbf3ab5d59d23ff93cea307ef1fa074"} Sep 05 06:28:48.761862 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:48.761833 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cmxtg_ffad8c9d-5409-4c11-a250-686bb69099d4/serve-healthcheck-canary/0.log" Sep 05 06:28:48.918735 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:48.918700 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-q8tzt" event={"ID":"01815101-b04b-43ea-bb6c-bc896fd4b969","Type":"ContainerStarted","Data":"d2071a45d4a8f207ee54a0905ad378698c2fe24305ec33200273c6469c6a2264"} Sep 05 06:28:55.036600 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:55.036573 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:28:55.040809 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:28:55.040787 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7c4df55d4f-tw9lb" Sep 05 06:29:07.661793 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:07.661690 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5457d4cb6c-lv5kj" podUID="3f53c51f-6fcb-4241-ad84-28d96684b3e6" containerName="console" containerID="cri-o://72a3891886c8130ef81257dc4ef3838998fa9635643631f27cc1925f673339e4" gracePeriod=15 Sep 05 06:29:07.914421 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:07.914366 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:29:07.999981 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:07.999946 2565 generic.go:356] "Generic (PLEG): container finished" podID="3f53c51f-6fcb-4241-ad84-28d96684b3e6" containerID="72a3891886c8130ef81257dc4ef3838998fa9635643631f27cc1925f673339e4" exitCode=0 Sep 05 06:29:08.000179 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:07.999993 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5457d4cb6c-lv5kj" Sep 05 06:29:08.000179 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:08.000039 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5457d4cb6c-lv5kj" event={"ID":"3f53c51f-6fcb-4241-ad84-28d96684b3e6","Type":"ContainerDied","Data":"72a3891886c8130ef81257dc4ef3838998fa9635643631f27cc1925f673339e4"} Sep 05 06:29:08.000179 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:08.000128 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5457d4cb6c-lv5kj" event={"ID":"3f53c51f-6fcb-4241-ad84-28d96684b3e6","Type":"ContainerDied","Data":"f912789af287bc91768d2a06c5765c551aeaad5b08b0addcf61485f36d278c6a"} Sep 05 06:29:08.000179 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:08.000158 2565 scope.go:117] "RemoveContainer" containerID="72a3891886c8130ef81257dc4ef3838998fa9635643631f27cc1925f673339e4" Sep 05 06:29:08.003445 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:08.003427 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f53c51f-6fcb-4241-ad84-28d96684b3e6-console-oauth-config\") pod \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\" (UID: \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\") " Sep 05 06:29:08.003549 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:08.003465 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f53c51f-6fcb-4241-ad84-28d96684b3e6-console-config\") pod \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\" (UID: \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\") " Sep 05 06:29:08.003549 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:08.003501 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrjm5\" (UniqueName: \"kubernetes.io/projected/3f53c51f-6fcb-4241-ad84-28d96684b3e6-kube-api-access-vrjm5\") pod \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\" (UID: \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\") " Sep 05 06:29:08.003549 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:08.003530 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f53c51f-6fcb-4241-ad84-28d96684b3e6-service-ca\") pod \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\" (UID: \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\") " Sep 05 06:29:08.003702 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:08.003569 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f53c51f-6fcb-4241-ad84-28d96684b3e6-oauth-serving-cert\") pod \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\" (UID: \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\") " Sep 05 06:29:08.003702 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:08.003599 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f53c51f-6fcb-4241-ad84-28d96684b3e6-console-serving-cert\") pod \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\" (UID: \"3f53c51f-6fcb-4241-ad84-28d96684b3e6\") " Sep 05 06:29:08.003950 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:08.003904 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f53c51f-6fcb-4241-ad84-28d96684b3e6-console-config" (OuterVolumeSpecName: "console-config") pod "3f53c51f-6fcb-4241-ad84-28d96684b3e6" (UID: "3f53c51f-6fcb-4241-ad84-28d96684b3e6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:29:08.004066 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:08.003941 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f53c51f-6fcb-4241-ad84-28d96684b3e6-service-ca" (OuterVolumeSpecName: "service-ca") pod "3f53c51f-6fcb-4241-ad84-28d96684b3e6" (UID: "3f53c51f-6fcb-4241-ad84-28d96684b3e6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:29:08.004066 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:08.003987 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f53c51f-6fcb-4241-ad84-28d96684b3e6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3f53c51f-6fcb-4241-ad84-28d96684b3e6" (UID: "3f53c51f-6fcb-4241-ad84-28d96684b3e6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:29:08.006180 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:08.006155 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f53c51f-6fcb-4241-ad84-28d96684b3e6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3f53c51f-6fcb-4241-ad84-28d96684b3e6" (UID: "3f53c51f-6fcb-4241-ad84-28d96684b3e6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 06:29:08.006434 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:08.006406 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f53c51f-6fcb-4241-ad84-28d96684b3e6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3f53c51f-6fcb-4241-ad84-28d96684b3e6" (UID: "3f53c51f-6fcb-4241-ad84-28d96684b3e6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 06:29:08.006434 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:08.006417 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f53c51f-6fcb-4241-ad84-28d96684b3e6-kube-api-access-vrjm5" (OuterVolumeSpecName: "kube-api-access-vrjm5") pod "3f53c51f-6fcb-4241-ad84-28d96684b3e6" (UID: "3f53c51f-6fcb-4241-ad84-28d96684b3e6"). InnerVolumeSpecName "kube-api-access-vrjm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:29:08.008986 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:08.008969 2565 scope.go:117] "RemoveContainer" containerID="72a3891886c8130ef81257dc4ef3838998fa9635643631f27cc1925f673339e4" Sep 05 06:29:08.009292 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:29:08.009275 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72a3891886c8130ef81257dc4ef3838998fa9635643631f27cc1925f673339e4\": container with ID starting with 72a3891886c8130ef81257dc4ef3838998fa9635643631f27cc1925f673339e4 not found: ID does not exist" containerID="72a3891886c8130ef81257dc4ef3838998fa9635643631f27cc1925f673339e4" Sep 05 06:29:08.009356 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:08.009300 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72a3891886c8130ef81257dc4ef3838998fa9635643631f27cc1925f673339e4"} err="failed to get container status \"72a3891886c8130ef81257dc4ef3838998fa9635643631f27cc1925f673339e4\": rpc error: code = NotFound desc = could not find container \"72a3891886c8130ef81257dc4ef3838998fa9635643631f27cc1925f673339e4\": container with ID starting with 72a3891886c8130ef81257dc4ef3838998fa9635643631f27cc1925f673339e4 not found: ID does not exist" Sep 05 06:29:08.104182 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:08.104141 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f53c51f-6fcb-4241-ad84-28d96684b3e6-console-oauth-config\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:29:08.104182 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:08.104173 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f53c51f-6fcb-4241-ad84-28d96684b3e6-console-config\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:29:08.104182 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:08.104182 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrjm5\" (UniqueName: \"kubernetes.io/projected/3f53c51f-6fcb-4241-ad84-28d96684b3e6-kube-api-access-vrjm5\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:29:08.104182 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:08.104190 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f53c51f-6fcb-4241-ad84-28d96684b3e6-service-ca\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:29:08.104482 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:08.104198 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f53c51f-6fcb-4241-ad84-28d96684b3e6-oauth-serving-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:29:08.104482 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:08.104206 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f53c51f-6fcb-4241-ad84-28d96684b3e6-console-serving-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:29:08.316737 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:08.316707 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5457d4cb6c-lv5kj"] Sep 05 06:29:08.320888 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:08.320864 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5457d4cb6c-lv5kj"] Sep 05 06:29:10.164785 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:10.164753 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f53c51f-6fcb-4241-ad84-28d96684b3e6" path="/var/lib/kubelet/pods/3f53c51f-6fcb-4241-ad84-28d96684b3e6/volumes" Sep 05 06:29:29.311704 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.311675 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55fb6f6765-shtc4"] Sep 05 06:29:29.312162 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.312000 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f53c51f-6fcb-4241-ad84-28d96684b3e6" containerName="console" Sep 05 06:29:29.312162 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.312013 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f53c51f-6fcb-4241-ad84-28d96684b3e6" containerName="console" Sep 05 06:29:29.312162 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.312066 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f53c51f-6fcb-4241-ad84-28d96684b3e6" containerName="console" Sep 05 06:29:29.316767 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.316747 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:29.319266 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.319245 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 05 06:29:29.319401 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.319249 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 05 06:29:29.319401 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.319268 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 05 06:29:29.319654 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.319634 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 05 06:29:29.319739 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.319676 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 05 06:29:29.319739 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.319730 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 05 06:29:29.319831 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.319730 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 05 06:29:29.319831 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.319809 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7r4f6\"" Sep 05 06:29:29.324944 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.324909 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 05 06:29:29.327134 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.327112 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55fb6f6765-shtc4"] Sep 05 06:29:29.365710 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.365672 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k656f\" (UniqueName: \"kubernetes.io/projected/a73a163e-4a37-4f39-8b60-6bc648a0d694-kube-api-access-k656f\") pod \"console-55fb6f6765-shtc4\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:29.365860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.365725 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a73a163e-4a37-4f39-8b60-6bc648a0d694-trusted-ca-bundle\") pod \"console-55fb6f6765-shtc4\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:29.365860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.365756 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a73a163e-4a37-4f39-8b60-6bc648a0d694-console-oauth-config\") pod \"console-55fb6f6765-shtc4\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:29.365860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.365810 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a73a163e-4a37-4f39-8b60-6bc648a0d694-oauth-serving-cert\") pod \"console-55fb6f6765-shtc4\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:29.366019 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.365864 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a73a163e-4a37-4f39-8b60-6bc648a0d694-service-ca\") pod \"console-55fb6f6765-shtc4\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:29.366019 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.365902 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a73a163e-4a37-4f39-8b60-6bc648a0d694-console-serving-cert\") pod \"console-55fb6f6765-shtc4\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:29.366019 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.365937 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a73a163e-4a37-4f39-8b60-6bc648a0d694-console-config\") pod \"console-55fb6f6765-shtc4\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:29.467188 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.467146 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a73a163e-4a37-4f39-8b60-6bc648a0d694-trusted-ca-bundle\") pod \"console-55fb6f6765-shtc4\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:29.467188 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.467191 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a73a163e-4a37-4f39-8b60-6bc648a0d694-console-oauth-config\") pod \"console-55fb6f6765-shtc4\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:29.467471 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.467231 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a73a163e-4a37-4f39-8b60-6bc648a0d694-oauth-serving-cert\") pod \"console-55fb6f6765-shtc4\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:29.467471 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.467270 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a73a163e-4a37-4f39-8b60-6bc648a0d694-service-ca\") pod \"console-55fb6f6765-shtc4\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:29.467471 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.467300 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a73a163e-4a37-4f39-8b60-6bc648a0d694-console-serving-cert\") pod \"console-55fb6f6765-shtc4\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:29.467471 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.467356 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a73a163e-4a37-4f39-8b60-6bc648a0d694-console-config\") pod \"console-55fb6f6765-shtc4\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:29.467471 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.467395 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k656f\" (UniqueName: \"kubernetes.io/projected/a73a163e-4a37-4f39-8b60-6bc648a0d694-kube-api-access-k656f\") pod \"console-55fb6f6765-shtc4\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:29.468008 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.467980 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a73a163e-4a37-4f39-8b60-6bc648a0d694-service-ca\") pod \"console-55fb6f6765-shtc4\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:29.468323 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.468122 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a73a163e-4a37-4f39-8b60-6bc648a0d694-oauth-serving-cert\") pod \"console-55fb6f6765-shtc4\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:29.468323 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.468141 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a73a163e-4a37-4f39-8b60-6bc648a0d694-console-config\") pod \"console-55fb6f6765-shtc4\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:29.468323 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.468141 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a73a163e-4a37-4f39-8b60-6bc648a0d694-trusted-ca-bundle\") pod \"console-55fb6f6765-shtc4\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:29.470126 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.470094 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a73a163e-4a37-4f39-8b60-6bc648a0d694-console-oauth-config\") pod \"console-55fb6f6765-shtc4\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:29.470216 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.470101 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a73a163e-4a37-4f39-8b60-6bc648a0d694-console-serving-cert\") pod \"console-55fb6f6765-shtc4\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:29.478627 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.478604 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k656f\" (UniqueName: \"kubernetes.io/projected/a73a163e-4a37-4f39-8b60-6bc648a0d694-kube-api-access-k656f\") pod \"console-55fb6f6765-shtc4\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:29.628225 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.628188 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:29.903120 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:29.903039 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55fb6f6765-shtc4"] Sep 05 06:29:29.906954 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:29:29.906930 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73a163e_4a37_4f39_8b60_6bc648a0d694.slice/crio-6173200ece6d3bfa939ecea0f89206bd23c2c9622b3a1ce7f26e5306bc905021 WatchSource:0}: Error finding container 6173200ece6d3bfa939ecea0f89206bd23c2c9622b3a1ce7f26e5306bc905021: Status 404 returned error can't find the container with id 6173200ece6d3bfa939ecea0f89206bd23c2c9622b3a1ce7f26e5306bc905021 Sep 05 06:29:30.087074 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:30.087039 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55fb6f6765-shtc4" event={"ID":"a73a163e-4a37-4f39-8b60-6bc648a0d694","Type":"ContainerStarted","Data":"e02ccbc28bd30e4cfb6f03b23d17c515f15cd695a0cec5f3c8032bbb8d9bbd1b"} Sep 05 06:29:30.087232 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:30.087088 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55fb6f6765-shtc4" event={"ID":"a73a163e-4a37-4f39-8b60-6bc648a0d694","Type":"ContainerStarted","Data":"6173200ece6d3bfa939ecea0f89206bd23c2c9622b3a1ce7f26e5306bc905021"} Sep 05 06:29:30.107501 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:30.107451 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55fb6f6765-shtc4" podStartSLOduration=1.107435599 podStartE2EDuration="1.107435599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:29:30.10646854 +0000 UTC m=+194.491011666" watchObservedRunningTime="2026-09-05 06:29:30.107435599 +0000 UTC m=+194.491978723" Sep 05 06:29:39.629295 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:39.629260 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:39.629835 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:39.629338 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:39.634031 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:39.634008 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:29:40.129783 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:29:40.129758 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:30:36.467209 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:36.467178 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-p7t28"] Sep 05 06:30:36.470242 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:36.470222 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-p7t28" Sep 05 06:30:36.472788 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:36.472766 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 05 06:30:36.472788 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:36.472786 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 05 06:30:36.473419 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:36.473403 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-hcvsc\"" Sep 05 06:30:36.481299 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:36.481278 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-p7t28"] Sep 05 06:30:36.520209 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:36.520175 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thsnq\" (UniqueName: \"kubernetes.io/projected/3bd5a29d-75ab-4b6f-b450-1ec50ade5254-kube-api-access-thsnq\") pod \"cert-manager-webhook-6486ffd997-p7t28\" (UID: \"3bd5a29d-75ab-4b6f-b450-1ec50ade5254\") " pod="cert-manager/cert-manager-webhook-6486ffd997-p7t28" Sep 05 06:30:36.520390 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:36.520249 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bd5a29d-75ab-4b6f-b450-1ec50ade5254-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-p7t28\" (UID: \"3bd5a29d-75ab-4b6f-b450-1ec50ade5254\") " pod="cert-manager/cert-manager-webhook-6486ffd997-p7t28" Sep 05 06:30:36.621578 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:36.621489 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thsnq\" (UniqueName: \"kubernetes.io/projected/3bd5a29d-75ab-4b6f-b450-1ec50ade5254-kube-api-access-thsnq\") pod \"cert-manager-webhook-6486ffd997-p7t28\" (UID: \"3bd5a29d-75ab-4b6f-b450-1ec50ade5254\") " pod="cert-manager/cert-manager-webhook-6486ffd997-p7t28" Sep 05 06:30:36.621730 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:36.621593 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bd5a29d-75ab-4b6f-b450-1ec50ade5254-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-p7t28\" (UID: \"3bd5a29d-75ab-4b6f-b450-1ec50ade5254\") " pod="cert-manager/cert-manager-webhook-6486ffd997-p7t28" Sep 05 06:30:36.631571 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:36.631540 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bd5a29d-75ab-4b6f-b450-1ec50ade5254-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-p7t28\" (UID: \"3bd5a29d-75ab-4b6f-b450-1ec50ade5254\") " pod="cert-manager/cert-manager-webhook-6486ffd997-p7t28" Sep 05 06:30:36.631722 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:36.631654 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thsnq\" (UniqueName: \"kubernetes.io/projected/3bd5a29d-75ab-4b6f-b450-1ec50ade5254-kube-api-access-thsnq\") pod \"cert-manager-webhook-6486ffd997-p7t28\" (UID: \"3bd5a29d-75ab-4b6f-b450-1ec50ade5254\") " pod="cert-manager/cert-manager-webhook-6486ffd997-p7t28" Sep 05 06:30:36.788847 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:36.788745 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-p7t28" Sep 05 06:30:37.062960 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:37.062886 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-p7t28"] Sep 05 06:30:37.066651 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:30:37.066625 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bd5a29d_75ab_4b6f_b450_1ec50ade5254.slice/crio-0400b2242610365ebf7aa36b9e52261fcfadeb308d03b36fa8cc3830e570b5c6 WatchSource:0}: Error finding container 0400b2242610365ebf7aa36b9e52261fcfadeb308d03b36fa8cc3830e570b5c6: Status 404 returned error can't find the container with id 0400b2242610365ebf7aa36b9e52261fcfadeb308d03b36fa8cc3830e570b5c6 Sep 05 06:30:37.348478 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:37.348396 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-p7t28" event={"ID":"3bd5a29d-75ab-4b6f-b450-1ec50ade5254","Type":"ContainerStarted","Data":"0400b2242610365ebf7aa36b9e52261fcfadeb308d03b36fa8cc3830e570b5c6"} Sep 05 06:30:40.363907 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:40.363872 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-p7t28" event={"ID":"3bd5a29d-75ab-4b6f-b450-1ec50ade5254","Type":"ContainerStarted","Data":"ea882d19888f60ad7ec39913dea1e47ded44a38b6bd0d1b7b3266a0a0456ea65"} Sep 05 06:30:40.364400 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:40.363996 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-p7t28" Sep 05 06:30:40.386388 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:40.386338 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-p7t28" podStartSLOduration=1.405433785 podStartE2EDuration="4.386263169s" totalImagesPullingTime="2.980829384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:30:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:30:40.386111637 +0000 UTC m=+264.770654772" watchObservedRunningTime="2026-09-05 06:30:40.386263169 +0000 UTC m=+264.770806292" Sep 05 06:30:46.370118 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:46.370086 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-p7t28" Sep 05 06:30:46.896642 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:46.896608 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zbckx"] Sep 05 06:30:46.899880 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:46.899863 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zbckx" Sep 05 06:30:46.902300 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:46.902259 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 05 06:30:46.903076 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:46.903056 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-6tzvj\"" Sep 05 06:30:46.903173 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:46.903056 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 05 06:30:46.909843 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:46.909818 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zbckx"] Sep 05 06:30:47.012849 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:47.012818 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bc7cfc7-cf93-499e-a3fb-22c1b28fb662-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zbckx\" (UID: \"1bc7cfc7-cf93-499e-a3fb-22c1b28fb662\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zbckx" Sep 05 06:30:47.013023 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:47.012887 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4j5v\" (UniqueName: \"kubernetes.io/projected/1bc7cfc7-cf93-499e-a3fb-22c1b28fb662-kube-api-access-p4j5v\") pod \"openshift-lws-operator-fd8ccff4c-zbckx\" (UID: \"1bc7cfc7-cf93-499e-a3fb-22c1b28fb662\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zbckx" Sep 05 06:30:47.113704 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:47.113667 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bc7cfc7-cf93-499e-a3fb-22c1b28fb662-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zbckx\" (UID: \"1bc7cfc7-cf93-499e-a3fb-22c1b28fb662\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zbckx" Sep 05 06:30:47.113862 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:47.113745 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4j5v\" (UniqueName: \"kubernetes.io/projected/1bc7cfc7-cf93-499e-a3fb-22c1b28fb662-kube-api-access-p4j5v\") pod \"openshift-lws-operator-fd8ccff4c-zbckx\" (UID: \"1bc7cfc7-cf93-499e-a3fb-22c1b28fb662\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zbckx" Sep 05 06:30:47.114035 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:47.114016 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bc7cfc7-cf93-499e-a3fb-22c1b28fb662-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zbckx\" (UID: \"1bc7cfc7-cf93-499e-a3fb-22c1b28fb662\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zbckx" Sep 05 06:30:47.126402 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:47.126372 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4j5v\" (UniqueName: \"kubernetes.io/projected/1bc7cfc7-cf93-499e-a3fb-22c1b28fb662-kube-api-access-p4j5v\") pod \"openshift-lws-operator-fd8ccff4c-zbckx\" (UID: \"1bc7cfc7-cf93-499e-a3fb-22c1b28fb662\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zbckx" Sep 05 06:30:47.209610 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:47.209522 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zbckx" Sep 05 06:30:47.487943 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:47.487867 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zbckx"] Sep 05 06:30:47.491274 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:30:47.491249 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bc7cfc7_cf93_499e_a3fb_22c1b28fb662.slice/crio-a342063b6a3e334ff30f2de8e9809776dfdce779c486e7eae0496e2c0ff9e047 WatchSource:0}: Error finding container a342063b6a3e334ff30f2de8e9809776dfdce779c486e7eae0496e2c0ff9e047: Status 404 returned error can't find the container with id a342063b6a3e334ff30f2de8e9809776dfdce779c486e7eae0496e2c0ff9e047 Sep 05 06:30:48.395335 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:48.395279 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zbckx" event={"ID":"1bc7cfc7-cf93-499e-a3fb-22c1b28fb662","Type":"ContainerStarted","Data":"a342063b6a3e334ff30f2de8e9809776dfdce779c486e7eae0496e2c0ff9e047"} Sep 05 06:30:50.405802 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:50.405765 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zbckx" event={"ID":"1bc7cfc7-cf93-499e-a3fb-22c1b28fb662","Type":"ContainerStarted","Data":"a03ac7f83e70595ac09a639d52dc6157d9c29b0f1043d66fc03a31b7e67df2b6"} Sep 05 06:30:50.440759 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:30:50.440707 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zbckx" podStartSLOduration=2.117198205 podStartE2EDuration="4.440688793s" totalImagesPullingTime="2.323490588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:30:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:30:50.435604718 +0000 UTC m=+274.820147839" watchObservedRunningTime="2026-09-05 06:30:50.440688793 +0000 UTC m=+274.825231916" Sep 05 06:31:08.268861 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:08.268829 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx"] Sep 05 06:31:08.272188 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:08.272173 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx" Sep 05 06:31:08.275410 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:08.274733 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 05 06:31:08.275410 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:08.274986 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2f8p8\"" Sep 05 06:31:08.275410 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:08.275057 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 05 06:31:08.275410 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:08.275193 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 05 06:31:08.275410 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:08.275295 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 05 06:31:08.284817 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:08.284783 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx"] Sep 05 06:31:08.300433 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:08.300377 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68pgd\" (UniqueName: \"kubernetes.io/projected/d2803fb6-d850-4d46-9a7a-7b59edd9326c-kube-api-access-68pgd\") pod \"opendatahub-operator-controller-manager-555dc54ddd-f7khx\" (UID: \"d2803fb6-d850-4d46-9a7a-7b59edd9326c\") " pod="opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx" Sep 05 06:31:08.300433 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:08.300413 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d2803fb6-d850-4d46-9a7a-7b59edd9326c-manifests\") pod \"opendatahub-operator-controller-manager-555dc54ddd-f7khx\" (UID: \"d2803fb6-d850-4d46-9a7a-7b59edd9326c\") " pod="opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx" Sep 05 06:31:08.300433 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:08.300440 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d2803fb6-d850-4d46-9a7a-7b59edd9326c-webhook-cert\") pod \"opendatahub-operator-controller-manager-555dc54ddd-f7khx\" (UID: \"d2803fb6-d850-4d46-9a7a-7b59edd9326c\") " pod="opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx" Sep 05 06:31:08.300736 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:08.300470 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d2803fb6-d850-4d46-9a7a-7b59edd9326c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-555dc54ddd-f7khx\" (UID: \"d2803fb6-d850-4d46-9a7a-7b59edd9326c\") " pod="opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx" Sep 05 06:31:08.300736 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:08.300598 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2803fb6-d850-4d46-9a7a-7b59edd9326c-tmp\") pod \"opendatahub-operator-controller-manager-555dc54ddd-f7khx\" (UID: \"d2803fb6-d850-4d46-9a7a-7b59edd9326c\") " pod="opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx" Sep 05 06:31:08.401538 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:08.401504 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68pgd\" (UniqueName: \"kubernetes.io/projected/d2803fb6-d850-4d46-9a7a-7b59edd9326c-kube-api-access-68pgd\") pod \"opendatahub-operator-controller-manager-555dc54ddd-f7khx\" (UID: \"d2803fb6-d850-4d46-9a7a-7b59edd9326c\") " pod="opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx" Sep 05 06:31:08.401538 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:08.401543 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d2803fb6-d850-4d46-9a7a-7b59edd9326c-manifests\") pod \"opendatahub-operator-controller-manager-555dc54ddd-f7khx\" (UID: \"d2803fb6-d850-4d46-9a7a-7b59edd9326c\") " pod="opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx" Sep 05 06:31:08.401806 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:08.401560 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d2803fb6-d850-4d46-9a7a-7b59edd9326c-webhook-cert\") pod \"opendatahub-operator-controller-manager-555dc54ddd-f7khx\" (UID: \"d2803fb6-d850-4d46-9a7a-7b59edd9326c\") " pod="opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx" Sep 05 06:31:08.401806 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:08.401592 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d2803fb6-d850-4d46-9a7a-7b59edd9326c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-555dc54ddd-f7khx\" (UID: \"d2803fb6-d850-4d46-9a7a-7b59edd9326c\") " pod="opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx" Sep 05 06:31:08.401806 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:08.401661 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2803fb6-d850-4d46-9a7a-7b59edd9326c-tmp\") pod \"opendatahub-operator-controller-manager-555dc54ddd-f7khx\" (UID: \"d2803fb6-d850-4d46-9a7a-7b59edd9326c\") " pod="opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx" Sep 05 06:31:08.402011 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:08.401977 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d2803fb6-d850-4d46-9a7a-7b59edd9326c-manifests\") pod \"opendatahub-operator-controller-manager-555dc54ddd-f7khx\" (UID: \"d2803fb6-d850-4d46-9a7a-7b59edd9326c\") " pod="opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx" Sep 05 06:31:08.404004 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:08.403982 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2803fb6-d850-4d46-9a7a-7b59edd9326c-tmp\") pod \"opendatahub-operator-controller-manager-555dc54ddd-f7khx\" (UID: \"d2803fb6-d850-4d46-9a7a-7b59edd9326c\") " pod="opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx" Sep 05 06:31:08.404190 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:08.404170 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d2803fb6-d850-4d46-9a7a-7b59edd9326c-webhook-cert\") pod \"opendatahub-operator-controller-manager-555dc54ddd-f7khx\" (UID: \"d2803fb6-d850-4d46-9a7a-7b59edd9326c\") " pod="opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx" Sep 05 06:31:08.404262 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:08.404215 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d2803fb6-d850-4d46-9a7a-7b59edd9326c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-555dc54ddd-f7khx\" (UID: \"d2803fb6-d850-4d46-9a7a-7b59edd9326c\") " pod="opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx" Sep 05 06:31:08.422205 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:08.422175 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68pgd\" (UniqueName: \"kubernetes.io/projected/d2803fb6-d850-4d46-9a7a-7b59edd9326c-kube-api-access-68pgd\") pod \"opendatahub-operator-controller-manager-555dc54ddd-f7khx\" (UID: \"d2803fb6-d850-4d46-9a7a-7b59edd9326c\") " pod="opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx" Sep 05 06:31:08.588812 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:08.588722 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx" Sep 05 06:31:08.874339 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:08.874302 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx"] Sep 05 06:31:08.875846 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:31:08.875816 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2803fb6_d850_4d46_9a7a_7b59edd9326c.slice/crio-76f9be3a1b2cee9e3e751d81fccd558d34b7d9cbec19a3568fda6340d7a67400 WatchSource:0}: Error finding container 76f9be3a1b2cee9e3e751d81fccd558d34b7d9cbec19a3568fda6340d7a67400: Status 404 returned error can't find the container with id 76f9be3a1b2cee9e3e751d81fccd558d34b7d9cbec19a3568fda6340d7a67400 Sep 05 06:31:09.485175 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:09.485144 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx" event={"ID":"d2803fb6-d850-4d46-9a7a-7b59edd9326c","Type":"ContainerStarted","Data":"76f9be3a1b2cee9e3e751d81fccd558d34b7d9cbec19a3568fda6340d7a67400"} Sep 05 06:31:12.502273 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:12.502234 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx" event={"ID":"d2803fb6-d850-4d46-9a7a-7b59edd9326c","Type":"ContainerStarted","Data":"1a2aa90f9bb11ff3353f3fc42d965f447579c5e62bc6e19b191144ecba1d928c"} Sep 05 06:31:13.508746 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:13.508716 2565 generic.go:356] "Generic (PLEG): container finished" podID="d2803fb6-d850-4d46-9a7a-7b59edd9326c" containerID="1a2aa90f9bb11ff3353f3fc42d965f447579c5e62bc6e19b191144ecba1d928c" exitCode=0 Sep 05 06:31:13.509148 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:13.508750 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx" event={"ID":"d2803fb6-d850-4d46-9a7a-7b59edd9326c","Type":"ContainerDied","Data":"1a2aa90f9bb11ff3353f3fc42d965f447579c5e62bc6e19b191144ecba1d928c"} Sep 05 06:31:14.515718 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:14.515678 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx" event={"ID":"d2803fb6-d850-4d46-9a7a-7b59edd9326c","Type":"ContainerStarted","Data":"79747b1e61482f2d00286e420a8175190890e65f75bd17c1d4c29cc087e21bc9"} Sep 05 06:31:14.516190 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:14.515780 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx" Sep 05 06:31:14.541230 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:14.541176 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx" podStartSLOduration=1.923930798 podStartE2EDuration="6.541156875s" totalImagesPullingTime="3.585483513s" totalInitContainerRuntime="1.031742564s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:31:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:31:14.540778223 +0000 UTC m=+298.925321370" watchObservedRunningTime="2026-09-05 06:31:14.541156875 +0000 UTC m=+298.925700001" Sep 05 06:31:16.042767 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:16.042744 2565 kubelet.go:1648] "Image garbage collection succeeded" Sep 05 06:31:25.521786 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:25.521747 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-555dc54ddd-f7khx" Sep 05 06:31:26.372581 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:26.372546 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-66f494d4f4-vnfk5"] Sep 05 06:31:26.375016 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:26.374987 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-66f494d4f4-vnfk5" Sep 05 06:31:26.377382 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:26.377361 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 05 06:31:26.378218 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:26.378198 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 05 06:31:26.378293 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:26.378202 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 05 06:31:26.378378 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:26.378354 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tslgz\"" Sep 05 06:31:26.394920 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:26.394888 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-66f494d4f4-vnfk5"] Sep 05 06:31:26.465395 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:26.465356 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4-metrics-cert\") pod \"lws-controller-manager-66f494d4f4-vnfk5\" (UID: \"4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4\") " pod="openshift-lws-operator/lws-controller-manager-66f494d4f4-vnfk5" Sep 05 06:31:26.465550 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:26.465406 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2xhf\" (UniqueName: \"kubernetes.io/projected/4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4-kube-api-access-f2xhf\") pod \"lws-controller-manager-66f494d4f4-vnfk5\" (UID: \"4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4\") " pod="openshift-lws-operator/lws-controller-manager-66f494d4f4-vnfk5" Sep 05 06:31:26.465550 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:26.465484 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4-manager-config\") pod \"lws-controller-manager-66f494d4f4-vnfk5\" (UID: \"4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4\") " pod="openshift-lws-operator/lws-controller-manager-66f494d4f4-vnfk5" Sep 05 06:31:26.465624 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:26.465549 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4-cert\") pod \"lws-controller-manager-66f494d4f4-vnfk5\" (UID: \"4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4\") " pod="openshift-lws-operator/lws-controller-manager-66f494d4f4-vnfk5" Sep 05 06:31:26.566027 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:26.565992 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4-metrics-cert\") pod \"lws-controller-manager-66f494d4f4-vnfk5\" (UID: \"4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4\") " pod="openshift-lws-operator/lws-controller-manager-66f494d4f4-vnfk5" Sep 05 06:31:26.566527 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:26.566040 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2xhf\" (UniqueName: \"kubernetes.io/projected/4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4-kube-api-access-f2xhf\") pod \"lws-controller-manager-66f494d4f4-vnfk5\" (UID: \"4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4\") " pod="openshift-lws-operator/lws-controller-manager-66f494d4f4-vnfk5" Sep 05 06:31:26.566527 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:26.566076 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4-manager-config\") pod \"lws-controller-manager-66f494d4f4-vnfk5\" (UID: \"4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4\") " pod="openshift-lws-operator/lws-controller-manager-66f494d4f4-vnfk5" Sep 05 06:31:26.566527 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:26.566121 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4-cert\") pod \"lws-controller-manager-66f494d4f4-vnfk5\" (UID: \"4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4\") " pod="openshift-lws-operator/lws-controller-manager-66f494d4f4-vnfk5" Sep 05 06:31:26.566822 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:26.566796 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4-manager-config\") pod \"lws-controller-manager-66f494d4f4-vnfk5\" (UID: \"4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4\") " pod="openshift-lws-operator/lws-controller-manager-66f494d4f4-vnfk5" Sep 05 06:31:26.568654 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:26.568631 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4-metrics-cert\") pod \"lws-controller-manager-66f494d4f4-vnfk5\" (UID: \"4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4\") " pod="openshift-lws-operator/lws-controller-manager-66f494d4f4-vnfk5" Sep 05 06:31:26.568761 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:26.568670 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4-cert\") pod \"lws-controller-manager-66f494d4f4-vnfk5\" (UID: \"4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4\") " pod="openshift-lws-operator/lws-controller-manager-66f494d4f4-vnfk5" Sep 05 06:31:26.595879 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:26.595849 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2xhf\" (UniqueName: \"kubernetes.io/projected/4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4-kube-api-access-f2xhf\") pod \"lws-controller-manager-66f494d4f4-vnfk5\" (UID: \"4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4\") " pod="openshift-lws-operator/lws-controller-manager-66f494d4f4-vnfk5" Sep 05 06:31:26.684225 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:26.684135 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-66f494d4f4-vnfk5" Sep 05 06:31:26.955744 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:26.955537 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-66f494d4f4-vnfk5"] Sep 05 06:31:26.958488 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:31:26.958456 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac01a53_7ea0_4d67_8fb6_6e918f6c06e4.slice/crio-b4caa16177a3eb9df1f4a81350c8857f22912ecf9069f198428dd4a743538a55 WatchSource:0}: Error finding container b4caa16177a3eb9df1f4a81350c8857f22912ecf9069f198428dd4a743538a55: Status 404 returned error can't find the container with id b4caa16177a3eb9df1f4a81350c8857f22912ecf9069f198428dd4a743538a55 Sep 05 06:31:26.960492 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:26.960477 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 05 06:31:27.573266 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:27.573231 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-66f494d4f4-vnfk5" event={"ID":"4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4","Type":"ContainerStarted","Data":"b4caa16177a3eb9df1f4a81350c8857f22912ecf9069f198428dd4a743538a55"} Sep 05 06:31:29.583383 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:29.583344 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-66f494d4f4-vnfk5" event={"ID":"4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4","Type":"ContainerStarted","Data":"27960ade6b10d7fd8d18279521d7f9af4948e2493134e9bc1843248ae5ef1a3b"} Sep 05 06:31:29.583769 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:29.583425 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-66f494d4f4-vnfk5" Sep 05 06:31:29.604135 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:29.604086 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-66f494d4f4-vnfk5" podStartSLOduration=2.002772259 podStartE2EDuration="3.604071891s" totalImagesPullingTime="1.601299632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:31:29.603090672 +0000 UTC m=+313.987633796" watchObservedRunningTime="2026-09-05 06:31:29.604071891 +0000 UTC m=+313.988615014" Sep 05 06:31:35.208058 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.208010 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t"] Sep 05 06:31:35.211735 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.211714 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.214234 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.214213 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 05 06:31:35.214384 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.214232 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 05 06:31:35.214384 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.214221 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-kbjtg\"" Sep 05 06:31:35.214384 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.214372 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 05 06:31:35.223987 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.223965 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t"] Sep 05 06:31:35.349525 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.349484 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/632ec75c-4f06-4988-bb54-13e262319a5a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.349728 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.349553 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vrs2\" (UniqueName: \"kubernetes.io/projected/632ec75c-4f06-4988-bb54-13e262319a5a-kube-api-access-4vrs2\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.349728 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.349618 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/632ec75c-4f06-4988-bb54-13e262319a5a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.349728 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.349650 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/632ec75c-4f06-4988-bb54-13e262319a5a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.349728 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.349681 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/632ec75c-4f06-4988-bb54-13e262319a5a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.349971 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.349740 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/632ec75c-4f06-4988-bb54-13e262319a5a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.349971 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.349788 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/632ec75c-4f06-4988-bb54-13e262319a5a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.349971 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.349832 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/632ec75c-4f06-4988-bb54-13e262319a5a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.349971 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.349864 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/632ec75c-4f06-4988-bb54-13e262319a5a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.450792 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.450747 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vrs2\" (UniqueName: \"kubernetes.io/projected/632ec75c-4f06-4988-bb54-13e262319a5a-kube-api-access-4vrs2\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.450792 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.450798 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/632ec75c-4f06-4988-bb54-13e262319a5a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.451033 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.450821 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/632ec75c-4f06-4988-bb54-13e262319a5a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.451033 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.450841 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/632ec75c-4f06-4988-bb54-13e262319a5a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.451033 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.450865 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/632ec75c-4f06-4988-bb54-13e262319a5a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.451033 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.450889 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/632ec75c-4f06-4988-bb54-13e262319a5a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.451033 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.450908 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/632ec75c-4f06-4988-bb54-13e262319a5a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.451267 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.451125 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/632ec75c-4f06-4988-bb54-13e262319a5a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.451267 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.451194 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/632ec75c-4f06-4988-bb54-13e262319a5a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.451413 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.451296 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/632ec75c-4f06-4988-bb54-13e262319a5a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.451471 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.451438 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/632ec75c-4f06-4988-bb54-13e262319a5a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.451539 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.451517 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/632ec75c-4f06-4988-bb54-13e262319a5a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.451588 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.451537 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/632ec75c-4f06-4988-bb54-13e262319a5a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.451757 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.451741 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/632ec75c-4f06-4988-bb54-13e262319a5a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.454008 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.453982 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/632ec75c-4f06-4988-bb54-13e262319a5a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.454156 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.454137 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/632ec75c-4f06-4988-bb54-13e262319a5a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.461795 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.461729 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vrs2\" (UniqueName: \"kubernetes.io/projected/632ec75c-4f06-4988-bb54-13e262319a5a-kube-api-access-4vrs2\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.462357 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.462338 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/632ec75c-4f06-4988-bb54-13e262319a5a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799chzb4t\" (UID: \"632ec75c-4f06-4988-bb54-13e262319a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.521945 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.521917 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:35.804924 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:35.804899 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t"] Sep 05 06:31:35.807064 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:31:35.807037 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod632ec75c_4f06_4988_bb54_13e262319a5a.slice/crio-05fac22b75f497fdc4934f21d619315e1df1db88f939544f8bdc2410a4022e5d WatchSource:0}: Error finding container 05fac22b75f497fdc4934f21d619315e1df1db88f939544f8bdc2410a4022e5d: Status 404 returned error can't find the container with id 05fac22b75f497fdc4934f21d619315e1df1db88f939544f8bdc2410a4022e5d Sep 05 06:31:36.619500 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:36.619461 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" event={"ID":"632ec75c-4f06-4988-bb54-13e262319a5a","Type":"ContainerStarted","Data":"05fac22b75f497fdc4934f21d619315e1df1db88f939544f8bdc2410a4022e5d"} Sep 05 06:31:38.645286 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:38.645250 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-dcfx4"] Sep 05 06:31:38.652650 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:38.652618 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-dcfx4" Sep 05 06:31:38.655512 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:38.655489 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 05 06:31:38.655650 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:38.655497 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-wb8wg\"" Sep 05 06:31:38.659634 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:38.659612 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-dcfx4"] Sep 05 06:31:38.781109 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:38.781078 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3100836f-f0c6-4441-9dec-b4e06c85b0a0-cert\") pod \"odh-model-controller-7998c67dc4-dcfx4\" (UID: \"3100836f-f0c6-4441-9dec-b4e06c85b0a0\") " pod="opendatahub/odh-model-controller-7998c67dc4-dcfx4" Sep 05 06:31:38.781286 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:38.781128 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvcbs\" (UniqueName: \"kubernetes.io/projected/3100836f-f0c6-4441-9dec-b4e06c85b0a0-kube-api-access-kvcbs\") pod \"odh-model-controller-7998c67dc4-dcfx4\" (UID: \"3100836f-f0c6-4441-9dec-b4e06c85b0a0\") " pod="opendatahub/odh-model-controller-7998c67dc4-dcfx4" Sep 05 06:31:38.882184 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:38.882141 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3100836f-f0c6-4441-9dec-b4e06c85b0a0-cert\") pod \"odh-model-controller-7998c67dc4-dcfx4\" (UID: \"3100836f-f0c6-4441-9dec-b4e06c85b0a0\") " pod="opendatahub/odh-model-controller-7998c67dc4-dcfx4" Sep 05 06:31:38.882184 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:38.882184 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvcbs\" (UniqueName: \"kubernetes.io/projected/3100836f-f0c6-4441-9dec-b4e06c85b0a0-kube-api-access-kvcbs\") pod \"odh-model-controller-7998c67dc4-dcfx4\" (UID: \"3100836f-f0c6-4441-9dec-b4e06c85b0a0\") " pod="opendatahub/odh-model-controller-7998c67dc4-dcfx4" Sep 05 06:31:38.882447 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:31:38.882296 2565 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 05 06:31:38.882447 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:31:38.882392 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3100836f-f0c6-4441-9dec-b4e06c85b0a0-cert podName:3100836f-f0c6-4441-9dec-b4e06c85b0a0 nodeName:}" failed. No retries permitted until 2026-09-05 06:31:39.382373334 +0000 UTC m=+323.766916455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3100836f-f0c6-4441-9dec-b4e06c85b0a0-cert") pod "odh-model-controller-7998c67dc4-dcfx4" (UID: "3100836f-f0c6-4441-9dec-b4e06c85b0a0") : secret "odh-model-controller-webhook-cert" not found Sep 05 06:31:38.897008 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:38.896940 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvcbs\" (UniqueName: \"kubernetes.io/projected/3100836f-f0c6-4441-9dec-b4e06c85b0a0-kube-api-access-kvcbs\") pod \"odh-model-controller-7998c67dc4-dcfx4\" (UID: \"3100836f-f0c6-4441-9dec-b4e06c85b0a0\") " pod="opendatahub/odh-model-controller-7998c67dc4-dcfx4" Sep 05 06:31:38.965604 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:38.965561 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:31:38.965688 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:38.965644 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:31:38.965688 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:38.965673 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:31:39.386599 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:39.386564 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3100836f-f0c6-4441-9dec-b4e06c85b0a0-cert\") pod \"odh-model-controller-7998c67dc4-dcfx4\" (UID: \"3100836f-f0c6-4441-9dec-b4e06c85b0a0\") " pod="opendatahub/odh-model-controller-7998c67dc4-dcfx4" Sep 05 06:31:39.389262 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:39.389236 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3100836f-f0c6-4441-9dec-b4e06c85b0a0-cert\") pod \"odh-model-controller-7998c67dc4-dcfx4\" (UID: \"3100836f-f0c6-4441-9dec-b4e06c85b0a0\") " pod="opendatahub/odh-model-controller-7998c67dc4-dcfx4" Sep 05 06:31:39.566389 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:39.566345 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-dcfx4" Sep 05 06:31:39.638371 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:39.638241 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" event={"ID":"632ec75c-4f06-4988-bb54-13e262319a5a","Type":"ContainerStarted","Data":"d8e1cfb171081f905dd2fb71abe951a83ea295ec134c9c83b1726e299d8afb59"} Sep 05 06:31:39.662855 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:39.662797 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" podStartSLOduration=1.506493652 podStartE2EDuration="4.662780626s" totalImagesPullingTime="3.156286974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:31:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:31:39.660296668 +0000 UTC m=+324.044839791" watchObservedRunningTime="2026-09-05 06:31:39.662780626 +0000 UTC m=+324.047323749" Sep 05 06:31:39.847863 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:39.847836 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-dcfx4"] Sep 05 06:31:39.850175 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:31:39.850152 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3100836f_f0c6_4441_9dec_b4e06c85b0a0.slice/crio-16d9ef59e4b5d6a3ae2a23e3bb16d5796eb492d1c07a9c9e2eaf876378bcafa5 WatchSource:0}: Error finding container 16d9ef59e4b5d6a3ae2a23e3bb16d5796eb492d1c07a9c9e2eaf876378bcafa5: Status 404 returned error can't find the container with id 16d9ef59e4b5d6a3ae2a23e3bb16d5796eb492d1c07a9c9e2eaf876378bcafa5 Sep 05 06:31:40.522983 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:40.522897 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:40.529365 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:40.529334 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:40.590084 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:40.590047 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-66f494d4f4-vnfk5" Sep 05 06:31:40.643996 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:40.643963 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-dcfx4" event={"ID":"3100836f-f0c6-4441-9dec-b4e06c85b0a0","Type":"ContainerStarted","Data":"16d9ef59e4b5d6a3ae2a23e3bb16d5796eb492d1c07a9c9e2eaf876378bcafa5"} Sep 05 06:31:40.644182 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:40.644170 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:40.645294 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:40.645273 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799chzb4t" Sep 05 06:31:43.660963 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:43.660918 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-dcfx4" event={"ID":"3100836f-f0c6-4441-9dec-b4e06c85b0a0","Type":"ContainerStarted","Data":"82251e77c36d1f6e383f7e2a2ea10587ad91b2ec5911ffa77764d6012ec43845"} Sep 05 06:31:43.661358 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:43.661009 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-dcfx4" Sep 05 06:31:43.680371 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:43.680300 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-dcfx4" podStartSLOduration=2.684027499 podStartE2EDuration="5.680283748s" totalImagesPullingTime="2.996256249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:31:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:31:43.67837494 +0000 UTC m=+328.062918070" watchObservedRunningTime="2026-09-05 06:31:43.680283748 +0000 UTC m=+328.064826872" Sep 05 06:31:45.790856 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.790823 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr"] Sep 05 06:31:45.794476 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.794455 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" Sep 05 06:31:45.797183 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.797160 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 05 06:31:45.797183 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.797160 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 05 06:31:45.797394 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.797207 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wqmcp\"" Sep 05 06:31:45.797394 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.797237 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 05 06:31:45.808881 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.808859 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr"] Sep 05 06:31:45.836079 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.836052 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eea75e7c-6676-450b-9796-b531bf8d4120-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-zrvwr\" (UID: \"eea75e7c-6676-450b-9796-b531bf8d4120\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" Sep 05 06:31:45.836079 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.836090 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eea75e7c-6676-450b-9796-b531bf8d4120-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-zrvwr\" (UID: \"eea75e7c-6676-450b-9796-b531bf8d4120\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" Sep 05 06:31:45.836268 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.836116 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jfkp\" (UniqueName: \"kubernetes.io/projected/eea75e7c-6676-450b-9796-b531bf8d4120-kube-api-access-8jfkp\") pod \"istiod-openshift-gateway-5677ccb49b-zrvwr\" (UID: \"eea75e7c-6676-450b-9796-b531bf8d4120\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" Sep 05 06:31:45.836268 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.836188 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eea75e7c-6676-450b-9796-b531bf8d4120-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-zrvwr\" (UID: \"eea75e7c-6676-450b-9796-b531bf8d4120\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" Sep 05 06:31:45.836268 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.836231 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eea75e7c-6676-450b-9796-b531bf8d4120-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-zrvwr\" (UID: \"eea75e7c-6676-450b-9796-b531bf8d4120\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" Sep 05 06:31:45.836268 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.836258 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eea75e7c-6676-450b-9796-b531bf8d4120-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-zrvwr\" (UID: \"eea75e7c-6676-450b-9796-b531bf8d4120\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" Sep 05 06:31:45.836430 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.836280 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eea75e7c-6676-450b-9796-b531bf8d4120-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-zrvwr\" (UID: \"eea75e7c-6676-450b-9796-b531bf8d4120\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" Sep 05 06:31:45.936700 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.936666 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jfkp\" (UniqueName: \"kubernetes.io/projected/eea75e7c-6676-450b-9796-b531bf8d4120-kube-api-access-8jfkp\") pod \"istiod-openshift-gateway-5677ccb49b-zrvwr\" (UID: \"eea75e7c-6676-450b-9796-b531bf8d4120\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" Sep 05 06:31:45.936894 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.936739 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eea75e7c-6676-450b-9796-b531bf8d4120-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-zrvwr\" (UID: \"eea75e7c-6676-450b-9796-b531bf8d4120\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" Sep 05 06:31:45.936894 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.936773 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eea75e7c-6676-450b-9796-b531bf8d4120-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-zrvwr\" (UID: \"eea75e7c-6676-450b-9796-b531bf8d4120\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" Sep 05 06:31:45.936894 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.936798 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eea75e7c-6676-450b-9796-b531bf8d4120-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-zrvwr\" (UID: \"eea75e7c-6676-450b-9796-b531bf8d4120\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" Sep 05 06:31:45.937111 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.936932 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eea75e7c-6676-450b-9796-b531bf8d4120-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-zrvwr\" (UID: \"eea75e7c-6676-450b-9796-b531bf8d4120\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" Sep 05 06:31:45.937111 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.937003 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eea75e7c-6676-450b-9796-b531bf8d4120-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-zrvwr\" (UID: \"eea75e7c-6676-450b-9796-b531bf8d4120\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" Sep 05 06:31:45.937111 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.937048 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eea75e7c-6676-450b-9796-b531bf8d4120-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-zrvwr\" (UID: \"eea75e7c-6676-450b-9796-b531bf8d4120\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" Sep 05 06:31:45.937707 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.937682 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eea75e7c-6676-450b-9796-b531bf8d4120-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-zrvwr\" (UID: \"eea75e7c-6676-450b-9796-b531bf8d4120\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" Sep 05 06:31:45.939378 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.939358 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eea75e7c-6676-450b-9796-b531bf8d4120-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-zrvwr\" (UID: \"eea75e7c-6676-450b-9796-b531bf8d4120\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" Sep 05 06:31:45.939521 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.939504 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eea75e7c-6676-450b-9796-b531bf8d4120-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-zrvwr\" (UID: \"eea75e7c-6676-450b-9796-b531bf8d4120\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" Sep 05 06:31:45.939723 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.939707 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eea75e7c-6676-450b-9796-b531bf8d4120-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-zrvwr\" (UID: \"eea75e7c-6676-450b-9796-b531bf8d4120\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" Sep 05 06:31:45.939794 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.939758 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eea75e7c-6676-450b-9796-b531bf8d4120-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-zrvwr\" (UID: \"eea75e7c-6676-450b-9796-b531bf8d4120\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" Sep 05 06:31:45.947801 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.947777 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eea75e7c-6676-450b-9796-b531bf8d4120-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-zrvwr\" (UID: \"eea75e7c-6676-450b-9796-b531bf8d4120\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" Sep 05 06:31:45.948641 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:45.948616 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jfkp\" (UniqueName: \"kubernetes.io/projected/eea75e7c-6676-450b-9796-b531bf8d4120-kube-api-access-8jfkp\") pod \"istiod-openshift-gateway-5677ccb49b-zrvwr\" (UID: \"eea75e7c-6676-450b-9796-b531bf8d4120\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" Sep 05 06:31:46.104412 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:46.104386 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" Sep 05 06:31:46.391746 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:46.391722 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr"] Sep 05 06:31:46.393168 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:31:46.393142 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea75e7c_6676_450b_9796_b531bf8d4120.slice/crio-07eec8db19130403165e47b4d6f4fffd0a7f843efeb6785efb7ba0bdc1726cae WatchSource:0}: Error finding container 07eec8db19130403165e47b4d6f4fffd0a7f843efeb6785efb7ba0bdc1726cae: Status 404 returned error can't find the container with id 07eec8db19130403165e47b4d6f4fffd0a7f843efeb6785efb7ba0bdc1726cae Sep 05 06:31:46.677534 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:46.677446 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" event={"ID":"eea75e7c-6676-450b-9796-b531bf8d4120","Type":"ContainerStarted","Data":"07eec8db19130403165e47b4d6f4fffd0a7f843efeb6785efb7ba0bdc1726cae"} Sep 05 06:31:48.709701 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:48.709657 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:31:48.710074 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:48.709750 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:31:49.695927 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:49.695894 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" event={"ID":"eea75e7c-6676-450b-9796-b531bf8d4120","Type":"ContainerStarted","Data":"9bb0fe2f15f9cf545ea55ef4e012a59d88d073455a00a944a5660e85cefb0279"} Sep 05 06:31:49.696110 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:49.695965 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" Sep 05 06:31:49.697427 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:49.697400 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" Sep 05 06:31:49.728496 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:49.728450 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zrvwr" podStartSLOduration=2.4148645970000002 podStartE2EDuration="4.728433543s" totalImagesPullingTime="2.313568946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:31:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:31:49.727173726 +0000 UTC m=+334.111716852" watchObservedRunningTime="2026-09-05 06:31:49.728433543 +0000 UTC m=+334.112976667" Sep 05 06:31:54.668586 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:31:54.668553 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-dcfx4" Sep 05 06:33:36.082924 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:36.082888 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55fb6f6765-shtc4"] Sep 05 06:33:52.739707 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.739558 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q"] Sep 05 06:33:52.743258 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.743238 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.746092 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.746072 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-z4vdh\"" Sep 05 06:33:52.757945 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.757918 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q"] Sep 05 06:33:52.844505 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.844461 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f2706-fa69-4ca4-a14e-44897263f5bb-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.844505 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.844508 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405f2706-fa69-4ca4-a14e-44897263f5bb-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.844750 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.844578 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405f2706-fa69-4ca4-a14e-44897263f5bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.844750 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.844595 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb5rt\" (UniqueName: \"kubernetes.io/projected/405f2706-fa69-4ca4-a14e-44897263f5bb-kube-api-access-qb5rt\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.844750 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.844660 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405f2706-fa69-4ca4-a14e-44897263f5bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.844750 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.844676 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405f2706-fa69-4ca4-a14e-44897263f5bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.844750 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.844708 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405f2706-fa69-4ca4-a14e-44897263f5bb-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.844750 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.844730 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405f2706-fa69-4ca4-a14e-44897263f5bb-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.845080 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.844757 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f2706-fa69-4ca4-a14e-44897263f5bb-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.945958 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.945910 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405f2706-fa69-4ca4-a14e-44897263f5bb-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.946152 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.945979 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405f2706-fa69-4ca4-a14e-44897263f5bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.946152 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.946004 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb5rt\" (UniqueName: \"kubernetes.io/projected/405f2706-fa69-4ca4-a14e-44897263f5bb-kube-api-access-qb5rt\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.946152 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.946068 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405f2706-fa69-4ca4-a14e-44897263f5bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.946152 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.946092 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405f2706-fa69-4ca4-a14e-44897263f5bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.946152 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.946123 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405f2706-fa69-4ca4-a14e-44897263f5bb-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.946477 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.946156 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405f2706-fa69-4ca4-a14e-44897263f5bb-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.946477 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.946188 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f2706-fa69-4ca4-a14e-44897263f5bb-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.946477 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.946225 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f2706-fa69-4ca4-a14e-44897263f5bb-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.946477 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.946425 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405f2706-fa69-4ca4-a14e-44897263f5bb-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.946675 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.946615 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f2706-fa69-4ca4-a14e-44897263f5bb-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.946844 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.946818 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f2706-fa69-4ca4-a14e-44897263f5bb-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.946844 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.946836 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405f2706-fa69-4ca4-a14e-44897263f5bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.947159 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.946849 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405f2706-fa69-4ca4-a14e-44897263f5bb-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.948596 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.948570 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405f2706-fa69-4ca4-a14e-44897263f5bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.948871 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.948853 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405f2706-fa69-4ca4-a14e-44897263f5bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.957276 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.957247 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405f2706-fa69-4ca4-a14e-44897263f5bb-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:52.957445 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:52.957408 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb5rt\" (UniqueName: \"kubernetes.io/projected/405f2706-fa69-4ca4-a14e-44897263f5bb-kube-api-access-qb5rt\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vc22q\" (UID: \"405f2706-fa69-4ca4-a14e-44897263f5bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:53.054627 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:53.054531 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:53.335041 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:53.335016 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q"] Sep 05 06:33:53.337653 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:33:53.337629 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405f2706_fa69_4ca4_a14e_44897263f5bb.slice/crio-1739eb53a555c8d1bc5225b07b199fae7946d465318abf28a3d7eedaa5615114 WatchSource:0}: Error finding container 1739eb53a555c8d1bc5225b07b199fae7946d465318abf28a3d7eedaa5615114: Status 404 returned error can't find the container with id 1739eb53a555c8d1bc5225b07b199fae7946d465318abf28a3d7eedaa5615114 Sep 05 06:33:53.339712 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:53.339684 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:33:53.339785 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:53.339747 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:33:53.339785 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:53.339775 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:33:54.257305 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:54.257265 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" event={"ID":"405f2706-fa69-4ca4-a14e-44897263f5bb","Type":"ContainerStarted","Data":"81c0d0dadcb4d588314b80b87ee0d3eca6f979abe9d05de23551e6149c1d37f7"} Sep 05 06:33:54.257999 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:54.257340 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" event={"ID":"405f2706-fa69-4ca4-a14e-44897263f5bb","Type":"ContainerStarted","Data":"1739eb53a555c8d1bc5225b07b199fae7946d465318abf28a3d7eedaa5615114"} Sep 05 06:33:55.055128 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:55.055095 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:55.059796 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:55.059775 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:55.083929 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:55.083876 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" podStartSLOduration=3.083829874 podStartE2EDuration="3.083829874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:33:54.279540312 +0000 UTC m=+458.664083438" watchObservedRunningTime="2026-09-05 06:33:55.083829874 +0000 UTC m=+459.468372998" Sep 05 06:33:55.261283 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:55.261250 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:33:55.262352 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:33:55.262332 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:34:01.109798 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:01.109727 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55fb6f6765-shtc4" podUID="a73a163e-4a37-4f39-8b60-6bc648a0d694" containerName="console" containerID="cri-o://e02ccbc28bd30e4cfb6f03b23d17c515f15cd695a0cec5f3c8032bbb8d9bbd1b" gracePeriod=15 Sep 05 06:34:01.291577 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:01.291547 2565 generic.go:356] "Generic (PLEG): container finished" podID="a73a163e-4a37-4f39-8b60-6bc648a0d694" containerID="e02ccbc28bd30e4cfb6f03b23d17c515f15cd695a0cec5f3c8032bbb8d9bbd1b" exitCode=0 Sep 05 06:34:01.291744 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:01.291629 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55fb6f6765-shtc4" event={"ID":"a73a163e-4a37-4f39-8b60-6bc648a0d694","Type":"ContainerDied","Data":"e02ccbc28bd30e4cfb6f03b23d17c515f15cd695a0cec5f3c8032bbb8d9bbd1b"} Sep 05 06:34:01.351003 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:01.350977 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:34:01.529589 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:01.529496 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k656f\" (UniqueName: \"kubernetes.io/projected/a73a163e-4a37-4f39-8b60-6bc648a0d694-kube-api-access-k656f\") pod \"a73a163e-4a37-4f39-8b60-6bc648a0d694\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " Sep 05 06:34:01.529589 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:01.529555 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a73a163e-4a37-4f39-8b60-6bc648a0d694-console-serving-cert\") pod \"a73a163e-4a37-4f39-8b60-6bc648a0d694\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " Sep 05 06:34:01.529809 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:01.529595 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a73a163e-4a37-4f39-8b60-6bc648a0d694-console-oauth-config\") pod \"a73a163e-4a37-4f39-8b60-6bc648a0d694\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " Sep 05 06:34:01.529809 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:01.529697 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a73a163e-4a37-4f39-8b60-6bc648a0d694-console-config\") pod \"a73a163e-4a37-4f39-8b60-6bc648a0d694\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " Sep 05 06:34:01.529809 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:01.529788 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a73a163e-4a37-4f39-8b60-6bc648a0d694-trusted-ca-bundle\") pod \"a73a163e-4a37-4f39-8b60-6bc648a0d694\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " Sep 05 06:34:01.529958 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:01.529822 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a73a163e-4a37-4f39-8b60-6bc648a0d694-service-ca\") pod \"a73a163e-4a37-4f39-8b60-6bc648a0d694\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " Sep 05 06:34:01.529958 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:01.529874 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a73a163e-4a37-4f39-8b60-6bc648a0d694-oauth-serving-cert\") pod \"a73a163e-4a37-4f39-8b60-6bc648a0d694\" (UID: \"a73a163e-4a37-4f39-8b60-6bc648a0d694\") " Sep 05 06:34:01.530562 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:01.530536 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a73a163e-4a37-4f39-8b60-6bc648a0d694-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a73a163e-4a37-4f39-8b60-6bc648a0d694" (UID: "a73a163e-4a37-4f39-8b60-6bc648a0d694"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:34:01.530719 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:01.530690 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a73a163e-4a37-4f39-8b60-6bc648a0d694-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a73a163e-4a37-4f39-8b60-6bc648a0d694" (UID: "a73a163e-4a37-4f39-8b60-6bc648a0d694"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:34:01.530789 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:01.530714 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a73a163e-4a37-4f39-8b60-6bc648a0d694-service-ca" (OuterVolumeSpecName: "service-ca") pod "a73a163e-4a37-4f39-8b60-6bc648a0d694" (UID: "a73a163e-4a37-4f39-8b60-6bc648a0d694"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:34:01.531133 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:01.531094 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a73a163e-4a37-4f39-8b60-6bc648a0d694-console-config" (OuterVolumeSpecName: "console-config") pod "a73a163e-4a37-4f39-8b60-6bc648a0d694" (UID: "a73a163e-4a37-4f39-8b60-6bc648a0d694"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:34:01.532386 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:01.532357 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a73a163e-4a37-4f39-8b60-6bc648a0d694-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a73a163e-4a37-4f39-8b60-6bc648a0d694" (UID: "a73a163e-4a37-4f39-8b60-6bc648a0d694"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 06:34:01.532509 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:01.532448 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a73a163e-4a37-4f39-8b60-6bc648a0d694-kube-api-access-k656f" (OuterVolumeSpecName: "kube-api-access-k656f") pod "a73a163e-4a37-4f39-8b60-6bc648a0d694" (UID: "a73a163e-4a37-4f39-8b60-6bc648a0d694"). InnerVolumeSpecName "kube-api-access-k656f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:34:01.532586 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:01.532567 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a73a163e-4a37-4f39-8b60-6bc648a0d694-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a73a163e-4a37-4f39-8b60-6bc648a0d694" (UID: "a73a163e-4a37-4f39-8b60-6bc648a0d694"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 06:34:01.631168 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:01.631131 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a73a163e-4a37-4f39-8b60-6bc648a0d694-console-config\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:34:01.631168 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:01.631160 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a73a163e-4a37-4f39-8b60-6bc648a0d694-trusted-ca-bundle\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:34:01.631168 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:01.631173 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a73a163e-4a37-4f39-8b60-6bc648a0d694-service-ca\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:34:01.631432 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:01.631186 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a73a163e-4a37-4f39-8b60-6bc648a0d694-oauth-serving-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:34:01.631432 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:01.631197 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k656f\" (UniqueName: \"kubernetes.io/projected/a73a163e-4a37-4f39-8b60-6bc648a0d694-kube-api-access-k656f\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:34:01.631432 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:01.631209 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a73a163e-4a37-4f39-8b60-6bc648a0d694-console-serving-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:34:01.631432 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:01.631219 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a73a163e-4a37-4f39-8b60-6bc648a0d694-console-oauth-config\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:34:02.297670 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:02.297632 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55fb6f6765-shtc4" event={"ID":"a73a163e-4a37-4f39-8b60-6bc648a0d694","Type":"ContainerDied","Data":"6173200ece6d3bfa939ecea0f89206bd23c2c9622b3a1ce7f26e5306bc905021"} Sep 05 06:34:02.298109 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:02.297683 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55fb6f6765-shtc4" Sep 05 06:34:02.298109 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:02.297702 2565 scope.go:117] "RemoveContainer" containerID="e02ccbc28bd30e4cfb6f03b23d17c515f15cd695a0cec5f3c8032bbb8d9bbd1b" Sep 05 06:34:02.330750 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:02.330720 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55fb6f6765-shtc4"] Sep 05 06:34:02.341335 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:02.341281 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55fb6f6765-shtc4"] Sep 05 06:34:04.165984 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:34:04.165941 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a73a163e-4a37-4f39-8b60-6bc648a0d694" path="/var/lib/kubelet/pods/a73a163e-4a37-4f39-8b60-6bc648a0d694/volumes" Sep 05 06:35:22.524531 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:22.524495 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-2wvb8"] Sep 05 06:35:22.524939 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:22.524840 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a73a163e-4a37-4f39-8b60-6bc648a0d694" containerName="console" Sep 05 06:35:22.524939 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:22.524851 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a73a163e-4a37-4f39-8b60-6bc648a0d694" containerName="console" Sep 05 06:35:22.524939 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:22.524903 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a73a163e-4a37-4f39-8b60-6bc648a0d694" containerName="console" Sep 05 06:35:22.527678 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:22.527661 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2wvb8" Sep 05 06:35:22.529982 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:22.529963 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 05 06:35:22.529982 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:22.529979 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-pntf4\"" Sep 05 06:35:22.541471 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:22.541444 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-2wvb8"] Sep 05 06:35:22.643599 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:22.643563 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bplm\" (UniqueName: \"kubernetes.io/projected/a45a9276-9d3f-402d-9b68-4ffb4462be5b-kube-api-access-6bplm\") pod \"payload-pre-processing-59fbc47bf8-2wvb8\" (UID: \"a45a9276-9d3f-402d-9b68-4ffb4462be5b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2wvb8" Sep 05 06:35:22.643765 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:22.643687 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a45a9276-9d3f-402d-9b68-4ffb4462be5b-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-2wvb8\" (UID: \"a45a9276-9d3f-402d-9b68-4ffb4462be5b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2wvb8" Sep 05 06:35:22.745070 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:22.745036 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bplm\" (UniqueName: \"kubernetes.io/projected/a45a9276-9d3f-402d-9b68-4ffb4462be5b-kube-api-access-6bplm\") pod \"payload-pre-processing-59fbc47bf8-2wvb8\" (UID: \"a45a9276-9d3f-402d-9b68-4ffb4462be5b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2wvb8" Sep 05 06:35:22.745294 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:22.745125 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a45a9276-9d3f-402d-9b68-4ffb4462be5b-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-2wvb8\" (UID: \"a45a9276-9d3f-402d-9b68-4ffb4462be5b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2wvb8" Sep 05 06:35:22.745889 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:22.745862 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a45a9276-9d3f-402d-9b68-4ffb4462be5b-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-2wvb8\" (UID: \"a45a9276-9d3f-402d-9b68-4ffb4462be5b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2wvb8" Sep 05 06:35:22.763757 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:22.763726 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bplm\" (UniqueName: \"kubernetes.io/projected/a45a9276-9d3f-402d-9b68-4ffb4462be5b-kube-api-access-6bplm\") pod \"payload-pre-processing-59fbc47bf8-2wvb8\" (UID: \"a45a9276-9d3f-402d-9b68-4ffb4462be5b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2wvb8" Sep 05 06:35:22.837298 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:22.837216 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2wvb8" Sep 05 06:35:23.127145 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:23.127120 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-2wvb8"] Sep 05 06:35:23.130211 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:35:23.130184 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda45a9276_9d3f_402d_9b68_4ffb4462be5b.slice/crio-f60c1557d68b6e49530616bb361ea6d7e680aa2ae7e6e0afd4607e460f6cc0ef WatchSource:0}: Error finding container f60c1557d68b6e49530616bb361ea6d7e680aa2ae7e6e0afd4607e460f6cc0ef: Status 404 returned error can't find the container with id f60c1557d68b6e49530616bb361ea6d7e680aa2ae7e6e0afd4607e460f6cc0ef Sep 05 06:35:23.670939 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:23.670899 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2wvb8" event={"ID":"a45a9276-9d3f-402d-9b68-4ffb4462be5b","Type":"ContainerStarted","Data":"f60c1557d68b6e49530616bb361ea6d7e680aa2ae7e6e0afd4607e460f6cc0ef"} Sep 05 06:35:25.689862 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:25.689817 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2wvb8" event={"ID":"a45a9276-9d3f-402d-9b68-4ffb4462be5b","Type":"ContainerStarted","Data":"27a2cf5348cbc38caf42f6163c34226035c6ce61a5788475742374769f32ea2d"} Sep 05 06:35:25.690717 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:25.690687 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2wvb8" Sep 05 06:35:25.711723 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:25.711664 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2wvb8" podStartSLOduration=1.8695415579999999 podStartE2EDuration="3.711643181s" totalImagesPullingTime="1.842101623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:35:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:35:25.709978117 +0000 UTC m=+550.094521268" watchObservedRunningTime="2026-09-05 06:35:25.711643181 +0000 UTC m=+550.096186315" Sep 05 06:35:31.724496 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:31.724416 2565 generic.go:356] "Generic (PLEG): container finished" podID="405f2706-fa69-4ca4-a14e-44897263f5bb" containerID="81c0d0dadcb4d588314b80b87ee0d3eca6f979abe9d05de23551e6149c1d37f7" exitCode=0 Sep 05 06:35:31.724496 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:31.724455 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" event={"ID":"405f2706-fa69-4ca4-a14e-44897263f5bb","Type":"ContainerDied","Data":"81c0d0dadcb4d588314b80b87ee0d3eca6f979abe9d05de23551e6149c1d37f7"} Sep 05 06:35:31.724919 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:31.724846 2565 scope.go:117] "RemoveContainer" containerID="81c0d0dadcb4d588314b80b87ee0d3eca6f979abe9d05de23551e6149c1d37f7" Sep 05 06:35:31.725923 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:31.725899 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:35:31.725981 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:31.725961 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:35:31.726015 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:31.725995 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:35:32.730979 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:32.730944 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" event={"ID":"405f2706-fa69-4ca4-a14e-44897263f5bb","Type":"ContainerStarted","Data":"d675759b52cc585013f08ef49268c8f75b356606d0e335b093bb8418300c31b0"} Sep 05 06:35:33.055101 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:33.055022 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:35:33.055258 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:33.055232 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-vc22q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Sep 05 06:35:33.055407 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:33.055290 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" podUID="405f2706-fa69-4ca4-a14e-44897263f5bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Sep 05 06:35:34.055028 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:34.054992 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-vc22q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Sep 05 06:35:34.055452 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:34.055059 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" podUID="405f2706-fa69-4ca4-a14e-44897263f5bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Sep 05 06:35:35.055530 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:35.055491 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-vc22q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Sep 05 06:35:35.055990 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:35.055566 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" podUID="405f2706-fa69-4ca4-a14e-44897263f5bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Sep 05 06:35:36.055061 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:36.055025 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-vc22q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Sep 05 06:35:36.055235 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:36.055084 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" podUID="405f2706-fa69-4ca4-a14e-44897263f5bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Sep 05 06:35:37.055549 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:37.055509 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-vc22q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Sep 05 06:35:37.055933 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:37.055567 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" podUID="405f2706-fa69-4ca4-a14e-44897263f5bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Sep 05 06:35:37.706650 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:37.706621 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2wvb8" Sep 05 06:35:38.059721 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:38.059637 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:35:38.060116 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:38.059923 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:35:38.060897 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:38.060871 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vc22q" Sep 05 06:35:44.590247 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.590217 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms"] Sep 05 06:35:44.594557 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.594537 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" Sep 05 06:35:44.597089 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.597069 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 05 06:35:44.597711 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.597694 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jfksv\"" Sep 05 06:35:44.597818 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.597695 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 05 06:35:44.597818 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.597695 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 05 06:35:44.605698 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.605678 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms"] Sep 05 06:35:44.656033 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.656007 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms\" (UID: \"6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" Sep 05 06:35:44.656033 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.656039 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms\" (UID: \"6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" Sep 05 06:35:44.656236 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.656063 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms\" (UID: \"6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" Sep 05 06:35:44.656236 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.656098 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms\" (UID: \"6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" Sep 05 06:35:44.656236 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.656159 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv2jg\" (UniqueName: \"kubernetes.io/projected/6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48-kube-api-access-mv2jg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms\" (UID: \"6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" Sep 05 06:35:44.656236 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.656188 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms\" (UID: \"6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" Sep 05 06:35:44.757629 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.757125 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms\" (UID: \"6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" Sep 05 06:35:44.757896 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.757594 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms\" (UID: \"6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" Sep 05 06:35:44.758076 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.758041 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv2jg\" (UniqueName: \"kubernetes.io/projected/6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48-kube-api-access-mv2jg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms\" (UID: \"6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" Sep 05 06:35:44.758194 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.758118 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms\" (UID: \"6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" Sep 05 06:35:44.758194 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.758165 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms\" (UID: \"6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" Sep 05 06:35:44.758337 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.758197 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms\" (UID: \"6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" Sep 05 06:35:44.758337 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.758238 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms\" (UID: \"6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" Sep 05 06:35:44.758607 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.758587 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms\" (UID: \"6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" Sep 05 06:35:44.759223 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.759201 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms\" (UID: \"6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" Sep 05 06:35:44.761298 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.761275 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms\" (UID: \"6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" Sep 05 06:35:44.761468 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.761450 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms\" (UID: \"6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" Sep 05 06:35:44.766823 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.766804 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv2jg\" (UniqueName: \"kubernetes.io/projected/6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48-kube-api-access-mv2jg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms\" (UID: \"6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" Sep 05 06:35:44.905372 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:44.905342 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" Sep 05 06:35:45.183926 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:45.183905 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms"] Sep 05 06:35:45.186226 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:35:45.186197 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bfba1c2_a5ae_4fd7_bc6d_87d2808a1c48.slice/crio-ed3b290f42a1439ea6a7a0cc57779ff0214925c161aaa1f4b07a65deaeffe31e WatchSource:0}: Error finding container ed3b290f42a1439ea6a7a0cc57779ff0214925c161aaa1f4b07a65deaeffe31e: Status 404 returned error can't find the container with id ed3b290f42a1439ea6a7a0cc57779ff0214925c161aaa1f4b07a65deaeffe31e Sep 05 06:35:45.792829 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:45.792794 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" event={"ID":"6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48","Type":"ContainerStarted","Data":"ed3b290f42a1439ea6a7a0cc57779ff0214925c161aaa1f4b07a65deaeffe31e"} Sep 05 06:35:46.798600 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:46.798562 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" event={"ID":"6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48","Type":"ContainerStarted","Data":"4bfa8631038aa973b7e02d8af0bc48f5fbbc9132f20c039e97792df97c11f700"} Sep 05 06:35:46.818242 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:46.818192 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" podStartSLOduration=1.763464982 podStartE2EDuration="2.818172514s" totalImagesPullingTime="1.054707532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:35:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:35:46.816962847 +0000 UTC m=+571.201505984" watchObservedRunningTime="2026-09-05 06:35:46.818172514 +0000 UTC m=+571.202715638" Sep 05 06:35:54.905710 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:54.905666 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" Sep 05 06:35:54.906068 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:54.905744 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" Sep 05 06:35:54.918112 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:54.918090 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" Sep 05 06:35:55.857123 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:35:55.857094 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms" Sep 05 06:38:28.461353 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.461304 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v"] Sep 05 06:38:28.464766 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.464741 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.468065 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.468043 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-45e697-openshift-default-dockercfg-77d5j\"" Sep 05 06:38:28.492874 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.492847 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v"] Sep 05 06:38:28.543986 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.543953 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istio-envoy\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.544144 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.544000 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istio-data\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.544144 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.544055 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l9x8\" (UniqueName: \"kubernetes.io/projected/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-kube-api-access-5l9x8\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.544144 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.544111 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istio-podinfo\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.544251 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.544141 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istio-token\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.544251 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.544181 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istiod-ca-cert\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.544251 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.544207 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-credential-socket\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.544251 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.544241 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-workload-certs\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.544471 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.544373 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-workload-socket\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.645879 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.645849 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-credential-socket\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.646058 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.645889 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-workload-certs\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.646058 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.645935 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-workload-socket\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.646058 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.645958 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istio-envoy\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.646058 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.645975 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istio-data\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.646058 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.646005 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l9x8\" (UniqueName: \"kubernetes.io/projected/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-kube-api-access-5l9x8\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.646346 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.646107 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istio-podinfo\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.646346 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.646146 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istio-token\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.646346 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.646182 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istiod-ca-cert\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.646346 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.646279 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-credential-socket\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.646346 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.646337 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-workload-socket\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.646547 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.646386 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-workload-certs\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.646585 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.646550 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istio-data\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.646882 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.646862 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istiod-ca-cert\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.648424 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.648404 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istio-envoy\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.648685 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.648666 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istio-podinfo\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.655011 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.654981 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l9x8\" (UniqueName: \"kubernetes.io/projected/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-kube-api-access-5l9x8\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.655123 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.655105 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istio-token\") pod \"e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:28.774840 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:28.774745 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:29.055688 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:29.055632 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v"] Sep 05 06:38:29.057985 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:38:29.057959 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81fe750c_eb9c_4be0_9b1c_104d05da6b1d.slice/crio-2c0c739d3db82874bde3a376a7d7e579ea566fd2107fd05e5ca584bec7e9bd92 WatchSource:0}: Error finding container 2c0c739d3db82874bde3a376a7d7e579ea566fd2107fd05e5ca584bec7e9bd92: Status 404 returned error can't find the container with id 2c0c739d3db82874bde3a376a7d7e579ea566fd2107fd05e5ca584bec7e9bd92 Sep 05 06:38:29.060057 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:29.060022 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:38:29.060127 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:29.060092 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:38:29.060165 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:29.060125 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:38:29.554212 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:29.554177 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" event={"ID":"81fe750c-eb9c-4be0-9b1c-104d05da6b1d","Type":"ContainerStarted","Data":"4c69f78135dd03daa1f8c92a27a9d40febc5266b925d1fb7b76be28677c2f893"} Sep 05 06:38:29.554620 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:29.554235 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" event={"ID":"81fe750c-eb9c-4be0-9b1c-104d05da6b1d","Type":"ContainerStarted","Data":"2c0c739d3db82874bde3a376a7d7e579ea566fd2107fd05e5ca584bec7e9bd92"} Sep 05 06:38:29.579942 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:29.579893 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" podStartSLOduration=1.57987572 podStartE2EDuration="1.57987572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:38:29.57874086 +0000 UTC m=+733.963284021" watchObservedRunningTime="2026-09-05 06:38:29.57987572 +0000 UTC m=+733.964418841" Sep 05 06:38:29.775498 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:29.775465 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:30.780427 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:30.780392 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:31.563768 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:31.563743 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:31.564921 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:31.564903 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:33.670773 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:33.670742 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg"] Sep 05 06:38:33.674285 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:33.674264 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg" Sep 05 06:38:33.676746 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:33.676727 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-45e697\"" Sep 05 06:38:33.686022 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:33.686000 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg"] Sep 05 06:38:33.711153 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:33.711120 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p"] Sep 05 06:38:33.714500 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:33.714482 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p" Sep 05 06:38:33.717163 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:33.717141 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-45e697-dockercfg-6gx8d\"" Sep 05 06:38:33.725568 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:33.725546 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p"] Sep 05 06:38:33.797184 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:33.797148 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwrds\" (UniqueName: \"kubernetes.io/projected/ef869309-90f5-472c-ad55-035a0d36f774-kube-api-access-nwrds\") pod \"payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg\" (UID: \"ef869309-90f5-472c-ad55-035a0d36f774\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg" Sep 05 06:38:33.797184 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:33.797183 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpjn7\" (UniqueName: \"kubernetes.io/projected/01f57470-5045-42eb-91c1-1734be3eeadb-kube-api-access-fpjn7\") pod \"payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p\" (UID: \"01f57470-5045-42eb-91c1-1734be3eeadb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p" Sep 05 06:38:33.797462 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:33.797253 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef869309-90f5-472c-ad55-035a0d36f774-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg\" (UID: \"ef869309-90f5-472c-ad55-035a0d36f774\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg" Sep 05 06:38:33.797462 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:33.797271 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f57470-5045-42eb-91c1-1734be3eeadb-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p\" (UID: \"01f57470-5045-42eb-91c1-1734be3eeadb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p" Sep 05 06:38:33.898665 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:33.898623 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef869309-90f5-472c-ad55-035a0d36f774-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg\" (UID: \"ef869309-90f5-472c-ad55-035a0d36f774\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg" Sep 05 06:38:33.898665 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:33.898664 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f57470-5045-42eb-91c1-1734be3eeadb-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p\" (UID: \"01f57470-5045-42eb-91c1-1734be3eeadb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p" Sep 05 06:38:33.898867 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:33.898746 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwrds\" (UniqueName: \"kubernetes.io/projected/ef869309-90f5-472c-ad55-035a0d36f774-kube-api-access-nwrds\") pod \"payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg\" (UID: \"ef869309-90f5-472c-ad55-035a0d36f774\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg" Sep 05 06:38:33.898867 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:33.898761 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpjn7\" (UniqueName: \"kubernetes.io/projected/01f57470-5045-42eb-91c1-1734be3eeadb-kube-api-access-fpjn7\") pod \"payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p\" (UID: \"01f57470-5045-42eb-91c1-1734be3eeadb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p" Sep 05 06:38:33.899380 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:33.899360 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef869309-90f5-472c-ad55-035a0d36f774-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg\" (UID: \"ef869309-90f5-472c-ad55-035a0d36f774\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg" Sep 05 06:38:33.899543 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:33.899525 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f57470-5045-42eb-91c1-1734be3eeadb-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p\" (UID: \"01f57470-5045-42eb-91c1-1734be3eeadb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p" Sep 05 06:38:33.907845 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:33.907817 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpjn7\" (UniqueName: \"kubernetes.io/projected/01f57470-5045-42eb-91c1-1734be3eeadb-kube-api-access-fpjn7\") pod \"payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p\" (UID: \"01f57470-5045-42eb-91c1-1734be3eeadb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p" Sep 05 06:38:33.908038 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:33.908015 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwrds\" (UniqueName: \"kubernetes.io/projected/ef869309-90f5-472c-ad55-035a0d36f774-kube-api-access-nwrds\") pod \"payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg\" (UID: \"ef869309-90f5-472c-ad55-035a0d36f774\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg" Sep 05 06:38:33.986000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:33.985922 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg" Sep 05 06:38:34.025787 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:34.025750 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p" Sep 05 06:38:34.261941 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:34.261915 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg"] Sep 05 06:38:34.263568 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:38:34.263546 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef869309_90f5_472c_ad55_035a0d36f774.slice/crio-1ef8a494a2855a132f82c584286f374fbf32a3a6510e5b094f20d51f51e145f2 WatchSource:0}: Error finding container 1ef8a494a2855a132f82c584286f374fbf32a3a6510e5b094f20d51f51e145f2: Status 404 returned error can't find the container with id 1ef8a494a2855a132f82c584286f374fbf32a3a6510e5b094f20d51f51e145f2 Sep 05 06:38:34.309399 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:34.309376 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p"] Sep 05 06:38:34.311402 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:38:34.311376 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f57470_5045_42eb_91c1_1734be3eeadb.slice/crio-87ce7af81c27016ee352f4061cd6abe43a952689753f7f1003d4b382bc6abdc8 WatchSource:0}: Error finding container 87ce7af81c27016ee352f4061cd6abe43a952689753f7f1003d4b382bc6abdc8: Status 404 returned error can't find the container with id 87ce7af81c27016ee352f4061cd6abe43a952689753f7f1003d4b382bc6abdc8 Sep 05 06:38:34.581711 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:34.581609 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p" event={"ID":"01f57470-5045-42eb-91c1-1734be3eeadb","Type":"ContainerStarted","Data":"962340836d5c849da3a52f2ab2ee07b88b00d2285453626a096dd6610951391a"} Sep 05 06:38:34.581711 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:34.581660 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p" event={"ID":"01f57470-5045-42eb-91c1-1734be3eeadb","Type":"ContainerStarted","Data":"87ce7af81c27016ee352f4061cd6abe43a952689753f7f1003d4b382bc6abdc8"} Sep 05 06:38:34.581939 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:34.581805 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p" Sep 05 06:38:34.583522 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:34.583497 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg" event={"ID":"ef869309-90f5-472c-ad55-035a0d36f774","Type":"ContainerStarted","Data":"283d2b8b7de9597801ac739d9452fe2b6975ffacfe4eae056420793e56f6ff3e"} Sep 05 06:38:34.583632 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:34.583541 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg" event={"ID":"ef869309-90f5-472c-ad55-035a0d36f774","Type":"ContainerStarted","Data":"1ef8a494a2855a132f82c584286f374fbf32a3a6510e5b094f20d51f51e145f2"} Sep 05 06:38:34.583632 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:34.583574 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg" Sep 05 06:38:34.601876 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:34.601828 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p" podStartSLOduration=1.601810828 podStartE2EDuration="1.601810828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:38:34.600237832 +0000 UTC m=+738.984780978" watchObservedRunningTime="2026-09-05 06:38:34.601810828 +0000 UTC m=+738.986353942" Sep 05 06:38:34.618280 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:34.618227 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg" podStartSLOduration=1.6182132839999999 podStartE2EDuration="1.618213284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:38:34.616941492 +0000 UTC m=+739.001484613" watchObservedRunningTime="2026-09-05 06:38:34.618213284 +0000 UTC m=+739.002756407" Sep 05 06:38:36.879861 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:36.879828 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk"] Sep 05 06:38:36.883444 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:36.883423 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:36.887002 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:36.886980 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-8956b5-openshift-default-dockercfg-mfknq\"" Sep 05 06:38:36.896759 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:36.896736 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk"] Sep 05 06:38:36.951830 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:36.951788 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh"] Sep 05 06:38:36.955252 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:36.955229 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:36.957761 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:36.957727 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b1d1c87d-openshift-default-dockercfg-jxvkq\"" Sep 05 06:38:36.966823 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:36.966800 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh"] Sep 05 06:38:37.030062 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.030024 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da2ee56d-6635-4935-8dd0-ead921efff47-istio-token\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.030062 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.030065 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da2ee56d-6635-4935-8dd0-ead921efff47-istiod-ca-cert\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.030261 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.030083 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdw6m\" (UniqueName: \"kubernetes.io/projected/da2ee56d-6635-4935-8dd0-ead921efff47-kube-api-access-gdw6m\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.030261 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.030111 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-credential-socket\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.030261 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.030215 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-workload-socket\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.030371 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.030265 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-istio-envoy\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.030371 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.030357 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da2ee56d-6635-4935-8dd0-ead921efff47-istio-podinfo\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.030429 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.030384 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-workload-certs\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.030429 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.030398 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-istio-data\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.131157 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.131079 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-istio-data\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.131157 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.131131 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-istio-envoy\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.131157 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.131151 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/053a09bd-1524-4db8-b014-373cfeaf9fbc-istio-podinfo\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.131390 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.131217 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-workload-socket\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.131390 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.131252 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-workload-certs\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.131390 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.131272 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-credential-socket\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.131390 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.131293 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-istio-data\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.131390 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.131346 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-workload-certs\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.131390 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.131365 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da2ee56d-6635-4935-8dd0-ead921efff47-istio-token\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.131390 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.131381 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdw6m\" (UniqueName: \"kubernetes.io/projected/da2ee56d-6635-4935-8dd0-ead921efff47-kube-api-access-gdw6m\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.131678 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.131516 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/053a09bd-1524-4db8-b014-373cfeaf9fbc-istiod-ca-cert\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.131678 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.131566 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da2ee56d-6635-4935-8dd0-ead921efff47-istiod-ca-cert\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.131678 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.131628 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-workload-socket\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.131678 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.131672 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-istio-envoy\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.131860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.131703 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/053a09bd-1524-4db8-b014-373cfeaf9fbc-istio-token\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.131860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.131727 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-istio-data\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.131860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.131751 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da2ee56d-6635-4935-8dd0-ead921efff47-istio-podinfo\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.131860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.131780 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-workload-certs\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.131860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.131815 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-credential-socket\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.131860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.131846 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjjhh\" (UniqueName: \"kubernetes.io/projected/053a09bd-1524-4db8-b014-373cfeaf9fbc-kube-api-access-sjjhh\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.132130 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.131995 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-workload-socket\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.132130 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.132099 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-credential-socket\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.132255 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.132235 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da2ee56d-6635-4935-8dd0-ead921efff47-istiod-ca-cert\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.133979 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.133960 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-istio-envoy\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.134238 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.134219 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da2ee56d-6635-4935-8dd0-ead921efff47-istio-podinfo\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.141091 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.141071 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdw6m\" (UniqueName: \"kubernetes.io/projected/da2ee56d-6635-4935-8dd0-ead921efff47-kube-api-access-gdw6m\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.141166 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.141130 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da2ee56d-6635-4935-8dd0-ead921efff47-istio-token\") pod \"e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.193325 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.193285 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:37.232846 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.232805 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/053a09bd-1524-4db8-b014-373cfeaf9fbc-istio-token\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.233030 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.232884 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjjhh\" (UniqueName: \"kubernetes.io/projected/053a09bd-1524-4db8-b014-373cfeaf9fbc-kube-api-access-sjjhh\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.233030 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.232911 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-istio-data\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.233030 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.232956 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-istio-envoy\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.233030 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.232998 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/053a09bd-1524-4db8-b014-373cfeaf9fbc-istio-podinfo\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.233194 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.233035 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-workload-socket\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.233194 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.233061 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-workload-certs\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.233194 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.233094 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-credential-socket\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.233194 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.233140 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/053a09bd-1524-4db8-b014-373cfeaf9fbc-istiod-ca-cert\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.233369 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.233349 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-istio-data\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.233657 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.233608 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-credential-socket\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.233657 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.233638 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-workload-socket\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.233842 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.233803 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-workload-certs\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.234108 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.234088 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/053a09bd-1524-4db8-b014-373cfeaf9fbc-istiod-ca-cert\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.235806 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.235787 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/053a09bd-1524-4db8-b014-373cfeaf9fbc-istio-podinfo\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.235897 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.235886 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-istio-envoy\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.245954 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.245732 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/053a09bd-1524-4db8-b014-373cfeaf9fbc-istio-token\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.247114 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.247094 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjjhh\" (UniqueName: \"kubernetes.io/projected/053a09bd-1524-4db8-b014-373cfeaf9fbc-kube-api-access-sjjhh\") pod \"e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.266473 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.266443 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:37.474198 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.474146 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk"] Sep 05 06:38:37.476822 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:38:37.476799 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda2ee56d_6635_4935_8dd0_ead921efff47.slice/crio-ecac8d3bb284bcd80ed8e12715b0fc2c4427835f0c3f145289308582a1e87c44 WatchSource:0}: Error finding container ecac8d3bb284bcd80ed8e12715b0fc2c4427835f0c3f145289308582a1e87c44: Status 404 returned error can't find the container with id ecac8d3bb284bcd80ed8e12715b0fc2c4427835f0c3f145289308582a1e87c44 Sep 05 06:38:37.479115 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.479082 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:38:37.479212 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.479157 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:38:37.479212 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.479199 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:38:37.543977 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.543955 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh"] Sep 05 06:38:37.546538 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:38:37.546511 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod053a09bd_1524_4db8_b014_373cfeaf9fbc.slice/crio-c5f69e51d925369453d252374c19b422b2e704370dc8117f769a07f081e18116 WatchSource:0}: Error finding container c5f69e51d925369453d252374c19b422b2e704370dc8117f769a07f081e18116: Status 404 returned error can't find the container with id c5f69e51d925369453d252374c19b422b2e704370dc8117f769a07f081e18116 Sep 05 06:38:37.548848 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.548805 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:38:37.548950 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.548892 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:38:37.548950 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.548932 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:38:37.601001 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.600974 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" event={"ID":"053a09bd-1524-4db8-b014-373cfeaf9fbc","Type":"ContainerStarted","Data":"c5f69e51d925369453d252374c19b422b2e704370dc8117f769a07f081e18116"} Sep 05 06:38:37.602733 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.602709 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" event={"ID":"da2ee56d-6635-4935-8dd0-ead921efff47","Type":"ContainerStarted","Data":"9ddca6f1b58df477d5840b409012f2a40c9cfc0a00820590228d6db5f467e193"} Sep 05 06:38:37.602858 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.602750 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" event={"ID":"da2ee56d-6635-4935-8dd0-ead921efff47","Type":"ContainerStarted","Data":"ecac8d3bb284bcd80ed8e12715b0fc2c4427835f0c3f145289308582a1e87c44"} Sep 05 06:38:37.625500 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:37.625443 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" podStartSLOduration=1.625423124 podStartE2EDuration="1.625423124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:38:37.623254528 +0000 UTC m=+742.007797676" watchObservedRunningTime="2026-09-05 06:38:37.625423124 +0000 UTC m=+742.009966248" Sep 05 06:38:38.194041 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:38.194009 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:38.199191 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:38.199168 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:38.610664 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:38.610633 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" event={"ID":"053a09bd-1524-4db8-b014-373cfeaf9fbc","Type":"ContainerStarted","Data":"0e4dd3b0b67b7206aa7306233f052050eac698ad24887f546854defbda990038"} Sep 05 06:38:38.611007 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:38.610984 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:38.612219 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:38.612194 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:38:38.631570 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:38.631513 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" podStartSLOduration=2.631493311 podStartE2EDuration="2.631493311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:38:38.630049721 +0000 UTC m=+743.014592847" watchObservedRunningTime="2026-09-05 06:38:38.631493311 +0000 UTC m=+743.016036434" Sep 05 06:38:39.267493 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:39.267455 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:39.272411 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:39.272385 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:39.616191 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:39.616159 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:39.616988 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:39.616973 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:38:43.020775 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:43.020740 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6"] Sep 05 06:38:43.024707 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:43.024678 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6" Sep 05 06:38:43.027531 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:43.027509 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-8956b5\"" Sep 05 06:38:43.039388 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:43.039365 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6"] Sep 05 06:38:43.215242 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:43.215191 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9bfab8-27f1-489b-927e-fcbc95b482c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6\" (UID: \"cc9bfab8-27f1-489b-927e-fcbc95b482c0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6" Sep 05 06:38:43.215441 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:43.215357 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp4l7\" (UniqueName: \"kubernetes.io/projected/cc9bfab8-27f1-489b-927e-fcbc95b482c0-kube-api-access-mp4l7\") pod \"payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6\" (UID: \"cc9bfab8-27f1-489b-927e-fcbc95b482c0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6" Sep 05 06:38:43.316040 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:43.315959 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9bfab8-27f1-489b-927e-fcbc95b482c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6\" (UID: \"cc9bfab8-27f1-489b-927e-fcbc95b482c0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6" Sep 05 06:38:43.316215 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:43.316055 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp4l7\" (UniqueName: \"kubernetes.io/projected/cc9bfab8-27f1-489b-927e-fcbc95b482c0-kube-api-access-mp4l7\") pod \"payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6\" (UID: \"cc9bfab8-27f1-489b-927e-fcbc95b482c0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6" Sep 05 06:38:43.316694 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:43.316671 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9bfab8-27f1-489b-927e-fcbc95b482c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6\" (UID: \"cc9bfab8-27f1-489b-927e-fcbc95b482c0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6" Sep 05 06:38:43.325835 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:43.325812 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp4l7\" (UniqueName: \"kubernetes.io/projected/cc9bfab8-27f1-489b-927e-fcbc95b482c0-kube-api-access-mp4l7\") pod \"payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6\" (UID: \"cc9bfab8-27f1-489b-927e-fcbc95b482c0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6" Sep 05 06:38:43.339724 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:43.339698 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6" Sep 05 06:38:43.641554 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:43.641490 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6"] Sep 05 06:38:43.643829 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:38:43.643800 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9bfab8_27f1_489b_927e_fcbc95b482c0.slice/crio-1b1c1be6a167a0b56028d8138b71d9e30394bf82d2f1dc602e15e41f7a1e1e24 WatchSource:0}: Error finding container 1b1c1be6a167a0b56028d8138b71d9e30394bf82d2f1dc602e15e41f7a1e1e24: Status 404 returned error can't find the container with id 1b1c1be6a167a0b56028d8138b71d9e30394bf82d2f1dc602e15e41f7a1e1e24 Sep 05 06:38:44.643040 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:44.643000 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6" event={"ID":"cc9bfab8-27f1-489b-927e-fcbc95b482c0","Type":"ContainerStarted","Data":"a0abbb94b64b3e7f637cf0663dd96f7ce80274d4a321756c88e4e86271740b35"} Sep 05 06:38:44.643417 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:44.643066 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6" event={"ID":"cc9bfab8-27f1-489b-927e-fcbc95b482c0","Type":"ContainerStarted","Data":"1b1c1be6a167a0b56028d8138b71d9e30394bf82d2f1dc602e15e41f7a1e1e24"} Sep 05 06:38:44.643417 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:44.643099 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6" Sep 05 06:38:44.660911 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:44.660864 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6" podStartSLOduration=1.660849848 podStartE2EDuration="1.660849848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:38:44.659345856 +0000 UTC m=+749.043888971" watchObservedRunningTime="2026-09-05 06:38:44.660849848 +0000 UTC m=+749.045392970" Sep 05 06:38:45.591707 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:45.591672 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p" Sep 05 06:38:45.591897 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:45.591759 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg" Sep 05 06:38:47.488881 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:47.488848 2565 patch_prober.go:28] interesting pod/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": read tcp 10.133.0.2:54762->10.133.0.33:15021: read: connection reset by peer" start-of-body= Sep 05 06:38:47.489267 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:47.488929 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" podUID="81fe750c-eb9c-4be0-9b1c-104d05da6b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": read tcp 10.133.0.2:54762->10.133.0.33:15021: read: connection reset by peer" Sep 05 06:38:47.502672 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:38:47.502644 2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@1-12359-0.service\": RecentStats: unable to find data in memory cache]" Sep 05 06:38:47.661968 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:47.661887 2565 generic.go:356] "Generic (PLEG): container finished" podID="81fe750c-eb9c-4be0-9b1c-104d05da6b1d" containerID="4c69f78135dd03daa1f8c92a27a9d40febc5266b925d1fb7b76be28677c2f893" exitCode=0 Sep 05 06:38:47.661968 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:47.661932 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" event={"ID":"81fe750c-eb9c-4be0-9b1c-104d05da6b1d","Type":"ContainerDied","Data":"4c69f78135dd03daa1f8c92a27a9d40febc5266b925d1fb7b76be28677c2f893"} Sep 05 06:38:47.662543 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:47.662522 2565 scope.go:117] "RemoveContainer" containerID="4c69f78135dd03daa1f8c92a27a9d40febc5266b925d1fb7b76be28677c2f893" Sep 05 06:38:47.663794 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:47.663761 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:38:47.663859 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:47.663836 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:38:47.663894 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:47.663868 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:38:48.668859 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:48.668827 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" event={"ID":"81fe750c-eb9c-4be0-9b1c-104d05da6b1d","Type":"ContainerStarted","Data":"85e342ffee928bbc4d3d5c3262a446b0d6b8665db7a0f36cc6af1921751fa984"} Sep 05 06:38:48.775028 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:48.774993 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:48.775250 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:48.775226 2565 patch_prober.go:28] interesting pod/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 05 06:38:48.775348 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:48.775276 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" podUID="81fe750c-eb9c-4be0-9b1c-104d05da6b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 05 06:38:49.775660 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:49.775627 2565 patch_prober.go:28] interesting pod/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 05 06:38:49.776213 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:49.776181 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" podUID="81fe750c-eb9c-4be0-9b1c-104d05da6b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 05 06:38:50.776046 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:50.775588 2565 patch_prober.go:28] interesting pod/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 05 06:38:50.776046 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:50.775658 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" podUID="81fe750c-eb9c-4be0-9b1c-104d05da6b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 05 06:38:51.775568 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:51.775522 2565 patch_prober.go:28] interesting pod/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 05 06:38:51.775792 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:51.775592 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" podUID="81fe750c-eb9c-4be0-9b1c-104d05da6b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 05 06:38:52.776145 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:52.776095 2565 patch_prober.go:28] interesting pod/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 05 06:38:52.776802 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:52.776747 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" podUID="81fe750c-eb9c-4be0-9b1c-104d05da6b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 05 06:38:53.776201 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:53.776158 2565 patch_prober.go:28] interesting pod/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 05 06:38:53.776845 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:53.776803 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" podUID="81fe750c-eb9c-4be0-9b1c-104d05da6b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 05 06:38:54.795729 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:54.795692 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:54.797364 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:54.797183 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:54.799578 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:54.799489 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:38:55.652419 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:55.652384 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6" Sep 05 06:38:58.104153 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.104122 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk"] Sep 05 06:38:58.106743 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.106724 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.109131 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.109108 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7080a4-openshift-default-dockercfg-qf9kf\"" Sep 05 06:38:58.119890 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.119868 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk"] Sep 05 06:38:58.174753 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.174724 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ba3167-1e93-4e94-8e9a-fc8d96997556-istio-token\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.174940 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.174787 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-workload-socket\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.174940 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.174845 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx8k2\" (UniqueName: \"kubernetes.io/projected/77ba3167-1e93-4e94-8e9a-fc8d96997556-kube-api-access-fx8k2\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.174940 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.174897 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-workload-certs\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.175051 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.174991 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-credential-socket\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.175051 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.175041 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-istio-envoy\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.175110 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.175071 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ba3167-1e93-4e94-8e9a-fc8d96997556-istio-podinfo\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.175110 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.175097 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ba3167-1e93-4e94-8e9a-fc8d96997556-istiod-ca-cert\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.175172 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.175159 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-istio-data\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.276465 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.276432 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ba3167-1e93-4e94-8e9a-fc8d96997556-istio-token\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.276606 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.276494 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-workload-socket\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.276606 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.276516 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx8k2\" (UniqueName: \"kubernetes.io/projected/77ba3167-1e93-4e94-8e9a-fc8d96997556-kube-api-access-fx8k2\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.276606 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.276538 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-workload-certs\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.276606 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.276585 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-credential-socket\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.276797 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.276620 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-istio-envoy\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.276797 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.276649 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ba3167-1e93-4e94-8e9a-fc8d96997556-istio-podinfo\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.276797 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.276677 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ba3167-1e93-4e94-8e9a-fc8d96997556-istiod-ca-cert\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.276797 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.276711 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-istio-data\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.276970 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.276857 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-workload-socket\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.277125 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.277105 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-workload-certs\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.277255 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.277228 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-credential-socket\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.277334 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.277238 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-istio-data\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.277543 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.277436 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ba3167-1e93-4e94-8e9a-fc8d96997556-istiod-ca-cert\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.278976 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.278959 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-istio-envoy\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.279080 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.279062 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ba3167-1e93-4e94-8e9a-fc8d96997556-istio-podinfo\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.285884 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.285863 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx8k2\" (UniqueName: \"kubernetes.io/projected/77ba3167-1e93-4e94-8e9a-fc8d96997556-kube-api-access-fx8k2\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.285979 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.285882 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ba3167-1e93-4e94-8e9a-fc8d96997556-istio-token\") pod \"e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.420452 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.420354 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:38:58.700247 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.700199 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk"] Sep 05 06:38:58.702451 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:38:58.702422 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ba3167_1e93_4e94_8e9a_fc8d96997556.slice/crio-a06f0c1e82e775d03edf3633a047f7556bfb9df475326519aea114fda0f0a748 WatchSource:0}: Error finding container a06f0c1e82e775d03edf3633a047f7556bfb9df475326519aea114fda0f0a748: Status 404 returned error can't find the container with id a06f0c1e82e775d03edf3633a047f7556bfb9df475326519aea114fda0f0a748 Sep 05 06:38:58.704707 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.704678 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:38:58.704757 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.704741 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:38:58.704790 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.704770 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:38:58.752087 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.752060 2565 generic.go:356] "Generic (PLEG): container finished" podID="da2ee56d-6635-4935-8dd0-ead921efff47" containerID="9ddca6f1b58df477d5840b409012f2a40c9cfc0a00820590228d6db5f467e193" exitCode=0 Sep 05 06:38:58.752200 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.752132 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" event={"ID":"da2ee56d-6635-4935-8dd0-ead921efff47","Type":"ContainerDied","Data":"9ddca6f1b58df477d5840b409012f2a40c9cfc0a00820590228d6db5f467e193"} Sep 05 06:38:58.752604 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.752586 2565 scope.go:117] "RemoveContainer" containerID="9ddca6f1b58df477d5840b409012f2a40c9cfc0a00820590228d6db5f467e193" Sep 05 06:38:58.753674 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.753645 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" event={"ID":"77ba3167-1e93-4e94-8e9a-fc8d96997556","Type":"ContainerStarted","Data":"a06f0c1e82e775d03edf3633a047f7556bfb9df475326519aea114fda0f0a748"} Sep 05 06:38:58.753856 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.753826 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:38:58.753931 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.753916 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:38:58.753986 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.753960 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:38:58.755373 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.755348 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" event={"ID":"053a09bd-1524-4db8-b014-373cfeaf9fbc","Type":"ContainerDied","Data":"0e4dd3b0b67b7206aa7306233f052050eac698ad24887f546854defbda990038"} Sep 05 06:38:58.755471 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.755304 2565 generic.go:356] "Generic (PLEG): container finished" podID="053a09bd-1524-4db8-b014-373cfeaf9fbc" containerID="0e4dd3b0b67b7206aa7306233f052050eac698ad24887f546854defbda990038" exitCode=0 Sep 05 06:38:58.755731 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.755716 2565 scope.go:117] "RemoveContainer" containerID="0e4dd3b0b67b7206aa7306233f052050eac698ad24887f546854defbda990038" Sep 05 06:38:58.756775 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.756753 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:38:58.756891 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.756848 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:38:58.756891 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:58.756883 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:38:59.763245 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:59.763215 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" event={"ID":"053a09bd-1524-4db8-b014-373cfeaf9fbc","Type":"ContainerStarted","Data":"9f144cfd168ec544d3a097ad9c393d3717844f841c8e381a1db3710070e6a1c1"} Sep 05 06:38:59.765274 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:59.765248 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" event={"ID":"da2ee56d-6635-4935-8dd0-ead921efff47","Type":"ContainerStarted","Data":"9d21ce782c25220f14d0ae53edeb07a9fee46104da1246e3a666e5b9470db879"} Sep 05 06:38:59.766911 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:59.766889 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" event={"ID":"77ba3167-1e93-4e94-8e9a-fc8d96997556","Type":"ContainerStarted","Data":"1210c2fc59f60dcb17380ad0abcca521d57b13a7fb102f63363c79fb44770cf8"} Sep 05 06:38:59.834284 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:38:59.834221 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" podStartSLOduration=1.834200367 podStartE2EDuration="1.834200367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:38:59.832864499 +0000 UTC m=+764.217407623" watchObservedRunningTime="2026-09-05 06:38:59.834200367 +0000 UTC m=+764.218743490" Sep 05 06:39:00.193744 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:00.193707 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:39:00.194026 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:00.193827 2565 patch_prober.go:28] interesting pod/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Sep 05 06:39:00.194197 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:00.194175 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" podUID="da2ee56d-6635-4935-8dd0-ead921efff47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Sep 05 06:39:00.267129 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:00.267098 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:39:00.267454 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:00.267430 2565 patch_prober.go:28] interesting pod/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 05 06:39:00.267554 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:00.267474 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" podUID="053a09bd-1524-4db8-b014-373cfeaf9fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 05 06:39:00.423208 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:00.423167 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:39:00.431410 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:00.431378 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:39:00.771948 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:00.771920 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:39:00.772842 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:00.772824 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:39:01.195801 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:01.195753 2565 patch_prober.go:28] interesting pod/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Sep 05 06:39:01.196012 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:01.195827 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" podUID="da2ee56d-6635-4935-8dd0-ead921efff47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Sep 05 06:39:01.267662 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:01.267628 2565 patch_prober.go:28] interesting pod/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 05 06:39:01.267890 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:01.267702 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" podUID="053a09bd-1524-4db8-b014-373cfeaf9fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 05 06:39:02.194277 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:02.194240 2565 patch_prober.go:28] interesting pod/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Sep 05 06:39:02.194929 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:02.194893 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" podUID="da2ee56d-6635-4935-8dd0-ead921efff47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Sep 05 06:39:02.267580 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:02.267547 2565 patch_prober.go:28] interesting pod/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 05 06:39:02.267760 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:02.267602 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" podUID="053a09bd-1524-4db8-b014-373cfeaf9fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 05 06:39:03.194758 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:03.194718 2565 patch_prober.go:28] interesting pod/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Sep 05 06:39:03.195738 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:03.195291 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" podUID="da2ee56d-6635-4935-8dd0-ead921efff47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Sep 05 06:39:03.267688 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:03.267652 2565 patch_prober.go:28] interesting pod/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 05 06:39:03.267856 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:03.267714 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" podUID="053a09bd-1524-4db8-b014-373cfeaf9fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 05 06:39:04.020335 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.019956 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl"] Sep 05 06:39:04.033462 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.033433 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl"] Sep 05 06:39:04.033681 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.033664 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n"] Sep 05 06:39:04.033857 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.033845 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl" Sep 05 06:39:04.037477 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.037457 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n" Sep 05 06:39:04.038445 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.038199 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7080a4\"" Sep 05 06:39:04.040274 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.040233 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-7080a4-dockercfg-8ggqk\"" Sep 05 06:39:04.063159 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.063131 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n"] Sep 05 06:39:04.159925 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.159887 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02147d7b-95d4-4690-8e7e-edd85db122e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl\" (UID: \"02147d7b-95d4-4690-8e7e-edd85db122e3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl" Sep 05 06:39:04.160254 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.160233 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2wsd\" (UniqueName: \"kubernetes.io/projected/7e61038e-bc4e-42a7-bdb1-4c18fc425388-kube-api-access-j2wsd\") pod \"payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n\" (UID: \"7e61038e-bc4e-42a7-bdb1-4c18fc425388\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n" Sep 05 06:39:04.160472 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.160455 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgc9s\" (UniqueName: \"kubernetes.io/projected/02147d7b-95d4-4690-8e7e-edd85db122e3-kube-api-access-xgc9s\") pod \"payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl\" (UID: \"02147d7b-95d4-4690-8e7e-edd85db122e3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl" Sep 05 06:39:04.160599 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.160587 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e61038e-bc4e-42a7-bdb1-4c18fc425388-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n\" (UID: \"7e61038e-bc4e-42a7-bdb1-4c18fc425388\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n" Sep 05 06:39:04.195335 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.194670 2565 patch_prober.go:28] interesting pod/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Sep 05 06:39:04.195335 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.194734 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" podUID="da2ee56d-6635-4935-8dd0-ead921efff47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Sep 05 06:39:04.262175 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.262132 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2wsd\" (UniqueName: \"kubernetes.io/projected/7e61038e-bc4e-42a7-bdb1-4c18fc425388-kube-api-access-j2wsd\") pod \"payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n\" (UID: \"7e61038e-bc4e-42a7-bdb1-4c18fc425388\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n" Sep 05 06:39:04.262391 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.262203 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgc9s\" (UniqueName: \"kubernetes.io/projected/02147d7b-95d4-4690-8e7e-edd85db122e3-kube-api-access-xgc9s\") pod \"payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl\" (UID: \"02147d7b-95d4-4690-8e7e-edd85db122e3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl" Sep 05 06:39:04.262391 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.262228 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e61038e-bc4e-42a7-bdb1-4c18fc425388-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n\" (UID: \"7e61038e-bc4e-42a7-bdb1-4c18fc425388\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n" Sep 05 06:39:04.262506 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.262388 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02147d7b-95d4-4690-8e7e-edd85db122e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl\" (UID: \"02147d7b-95d4-4690-8e7e-edd85db122e3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl" Sep 05 06:39:04.263081 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.263054 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02147d7b-95d4-4690-8e7e-edd85db122e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl\" (UID: \"02147d7b-95d4-4690-8e7e-edd85db122e3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl" Sep 05 06:39:04.263174 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.263102 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e61038e-bc4e-42a7-bdb1-4c18fc425388-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n\" (UID: \"7e61038e-bc4e-42a7-bdb1-4c18fc425388\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n" Sep 05 06:39:04.267091 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.267073 2565 patch_prober.go:28] interesting pod/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 05 06:39:04.267153 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.267117 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" podUID="053a09bd-1524-4db8-b014-373cfeaf9fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 05 06:39:04.270985 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.270937 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgc9s\" (UniqueName: \"kubernetes.io/projected/02147d7b-95d4-4690-8e7e-edd85db122e3-kube-api-access-xgc9s\") pod \"payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl\" (UID: \"02147d7b-95d4-4690-8e7e-edd85db122e3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl" Sep 05 06:39:04.271132 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.271110 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2wsd\" (UniqueName: \"kubernetes.io/projected/7e61038e-bc4e-42a7-bdb1-4c18fc425388-kube-api-access-j2wsd\") pod \"payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n\" (UID: \"7e61038e-bc4e-42a7-bdb1-4c18fc425388\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n" Sep 05 06:39:04.349893 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.349864 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl" Sep 05 06:39:04.355907 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.355885 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n" Sep 05 06:39:04.690165 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.690119 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl"] Sep 05 06:39:04.737071 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.737045 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n"] Sep 05 06:39:04.739102 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:39:04.739075 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e61038e_bc4e_42a7_bdb1_4c18fc425388.slice/crio-c0010576e774b7c58522af92a4577c1864a8024b46532c0f74d7f298412c4fbe WatchSource:0}: Error finding container c0010576e774b7c58522af92a4577c1864a8024b46532c0f74d7f298412c4fbe: Status 404 returned error can't find the container with id c0010576e774b7c58522af92a4577c1864a8024b46532c0f74d7f298412c4fbe Sep 05 06:39:04.854623 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.854583 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl" event={"ID":"02147d7b-95d4-4690-8e7e-edd85db122e3","Type":"ContainerStarted","Data":"2bd8bee13de2cb8473b08b98342b414000ff2e765b592c5030fbfe7c43b50af9"} Sep 05 06:39:04.854808 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.854649 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl" event={"ID":"02147d7b-95d4-4690-8e7e-edd85db122e3","Type":"ContainerStarted","Data":"1a56117547709524feb2f84c90ae322c0afc0dd1949748f4976af390e0a3f6b5"} Sep 05 06:39:04.854808 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.854700 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl" Sep 05 06:39:04.856392 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.856341 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n" event={"ID":"7e61038e-bc4e-42a7-bdb1-4c18fc425388","Type":"ContainerStarted","Data":"ae18dedf3d7c1d27b14e20b0a834a07dc7aaf69cbd983ee9027b556aa249c0a0"} Sep 05 06:39:04.856392 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.856390 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n" event={"ID":"7e61038e-bc4e-42a7-bdb1-4c18fc425388","Type":"ContainerStarted","Data":"c0010576e774b7c58522af92a4577c1864a8024b46532c0f74d7f298412c4fbe"} Sep 05 06:39:04.856597 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.856517 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n" Sep 05 06:39:04.879699 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.879620 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl" podStartSLOduration=1.8795969810000002 podStartE2EDuration="1.879596981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:39:04.877915245 +0000 UTC m=+769.262458379" watchObservedRunningTime="2026-09-05 06:39:04.879596981 +0000 UTC m=+769.264140105" Sep 05 06:39:04.899518 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:04.899448 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n" podStartSLOduration=0.899425024 podStartE2EDuration="899.425024ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:39:04.896390496 +0000 UTC m=+769.280933619" watchObservedRunningTime="2026-09-05 06:39:04.899425024 +0000 UTC m=+769.283968150" Sep 05 06:39:05.200684 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:05.200651 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:39:05.201570 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:05.201358 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:39:05.203095 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:05.203069 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:39:05.378756 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:05.378724 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:39:05.380002 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:05.379941 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:39:05.381190 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:05.381171 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:39:15.862870 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:15.862836 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n" Sep 05 06:39:15.863269 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:15.863016 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl" Sep 05 06:39:19.945862 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:19.945816 2565 generic.go:356] "Generic (PLEG): container finished" podID="77ba3167-1e93-4e94-8e9a-fc8d96997556" containerID="1210c2fc59f60dcb17380ad0abcca521d57b13a7fb102f63363c79fb44770cf8" exitCode=0 Sep 05 06:39:19.946427 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:19.945883 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" event={"ID":"77ba3167-1e93-4e94-8e9a-fc8d96997556","Type":"ContainerDied","Data":"1210c2fc59f60dcb17380ad0abcca521d57b13a7fb102f63363c79fb44770cf8"} Sep 05 06:39:19.946427 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:19.946390 2565 scope.go:117] "RemoveContainer" containerID="1210c2fc59f60dcb17380ad0abcca521d57b13a7fb102f63363c79fb44770cf8" Sep 05 06:39:19.947620 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:19.947576 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:39:19.947743 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:19.947652 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:39:19.947743 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:19.947704 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:39:20.953064 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:20.953028 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" event={"ID":"77ba3167-1e93-4e94-8e9a-fc8d96997556","Type":"ContainerStarted","Data":"d641554afba1f7ab537afd5d7fc8fc30090d60d429ef26eb6915da7002ece924"} Sep 05 06:39:21.421262 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:21.421221 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:39:21.421478 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:21.421444 2565 patch_prober.go:28] interesting pod/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 05 06:39:21.421525 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:21.421495 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" podUID="77ba3167-1e93-4e94-8e9a-fc8d96997556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 05 06:39:22.424073 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:22.424035 2565 patch_prober.go:28] interesting pod/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 05 06:39:22.424684 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:22.424652 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" podUID="77ba3167-1e93-4e94-8e9a-fc8d96997556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 05 06:39:22.737672 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:22.737581 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh"] Sep 05 06:39:22.738060 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:22.738030 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" podUID="053a09bd-1524-4db8-b014-373cfeaf9fbc" containerName="istio-proxy" containerID="cri-o://9f144cfd168ec544d3a097ad9c393d3717844f841c8e381a1db3710070e6a1c1" gracePeriod=30 Sep 05 06:39:23.421635 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:23.421603 2565 patch_prober.go:28] interesting pod/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 05 06:39:23.421825 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:23.421669 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" podUID="77ba3167-1e93-4e94-8e9a-fc8d96997556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 05 06:39:24.421022 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:24.420970 2565 patch_prober.go:28] interesting pod/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 05 06:39:24.421562 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:24.421062 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" podUID="77ba3167-1e93-4e94-8e9a-fc8d96997556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 05 06:39:25.421051 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:25.421020 2565 patch_prober.go:28] interesting pod/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 05 06:39:25.421456 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:25.421393 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" podUID="77ba3167-1e93-4e94-8e9a-fc8d96997556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 05 06:39:26.491586 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:26.491553 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:39:26.492058 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:26.491826 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:39:26.494242 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:26.494219 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:39:28.000016 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:27.999979 2565 generic.go:356] "Generic (PLEG): container finished" podID="053a09bd-1524-4db8-b014-373cfeaf9fbc" containerID="9f144cfd168ec544d3a097ad9c393d3717844f841c8e381a1db3710070e6a1c1" exitCode=0 Sep 05 06:39:28.000454 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.000060 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" event={"ID":"053a09bd-1524-4db8-b014-373cfeaf9fbc","Type":"ContainerDied","Data":"9f144cfd168ec544d3a097ad9c393d3717844f841c8e381a1db3710070e6a1c1"} Sep 05 06:39:28.000454 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.000132 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" event={"ID":"053a09bd-1524-4db8-b014-373cfeaf9fbc","Type":"ContainerDied","Data":"c5f69e51d925369453d252374c19b422b2e704370dc8117f769a07f081e18116"} Sep 05 06:39:28.000454 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.000155 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5f69e51d925369453d252374c19b422b2e704370dc8117f769a07f081e18116" Sep 05 06:39:28.000454 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.000174 2565 scope.go:117] "RemoveContainer" containerID="0e4dd3b0b67b7206aa7306233f052050eac698ad24887f546854defbda990038" Sep 05 06:39:28.010977 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.010955 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:39:28.138526 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.138480 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-workload-socket\") pod \"053a09bd-1524-4db8-b014-373cfeaf9fbc\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " Sep 05 06:39:28.138738 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.138558 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/053a09bd-1524-4db8-b014-373cfeaf9fbc-istiod-ca-cert\") pod \"053a09bd-1524-4db8-b014-373cfeaf9fbc\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " Sep 05 06:39:28.138738 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.138592 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-workload-certs\") pod \"053a09bd-1524-4db8-b014-373cfeaf9fbc\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " Sep 05 06:39:28.138738 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.138628 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-istio-data\") pod \"053a09bd-1524-4db8-b014-373cfeaf9fbc\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " Sep 05 06:39:28.138738 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.138669 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjjhh\" (UniqueName: \"kubernetes.io/projected/053a09bd-1524-4db8-b014-373cfeaf9fbc-kube-api-access-sjjhh\") pod \"053a09bd-1524-4db8-b014-373cfeaf9fbc\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " Sep 05 06:39:28.138738 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.138697 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-istio-envoy\") pod \"053a09bd-1524-4db8-b014-373cfeaf9fbc\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " Sep 05 06:39:28.138971 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.138747 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/053a09bd-1524-4db8-b014-373cfeaf9fbc-istio-token\") pod \"053a09bd-1524-4db8-b014-373cfeaf9fbc\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " Sep 05 06:39:28.138971 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.138794 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/053a09bd-1524-4db8-b014-373cfeaf9fbc-istio-podinfo\") pod \"053a09bd-1524-4db8-b014-373cfeaf9fbc\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " Sep 05 06:39:28.138971 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.138843 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-credential-socket\") pod \"053a09bd-1524-4db8-b014-373cfeaf9fbc\" (UID: \"053a09bd-1524-4db8-b014-373cfeaf9fbc\") " Sep 05 06:39:28.138971 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.138845 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "053a09bd-1524-4db8-b014-373cfeaf9fbc" (UID: "053a09bd-1524-4db8-b014-373cfeaf9fbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:39:28.138971 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.138857 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "053a09bd-1524-4db8-b014-373cfeaf9fbc" (UID: "053a09bd-1524-4db8-b014-373cfeaf9fbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:39:28.138971 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.138901 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-istio-data" (OuterVolumeSpecName: "istio-data") pod "053a09bd-1524-4db8-b014-373cfeaf9fbc" (UID: "053a09bd-1524-4db8-b014-373cfeaf9fbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:39:28.138971 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.138961 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/053a09bd-1524-4db8-b014-373cfeaf9fbc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "053a09bd-1524-4db8-b014-373cfeaf9fbc" (UID: "053a09bd-1524-4db8-b014-373cfeaf9fbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:39:28.139299 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.139114 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "053a09bd-1524-4db8-b014-373cfeaf9fbc" (UID: "053a09bd-1524-4db8-b014-373cfeaf9fbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:39:28.139442 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.139424 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-workload-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:39:28.139507 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.139443 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/053a09bd-1524-4db8-b014-373cfeaf9fbc-istiod-ca-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:39:28.139507 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.139452 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-workload-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:39:28.139507 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.139463 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-istio-data\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:39:28.139507 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.139474 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-credential-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:39:28.141456 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.141425 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "053a09bd-1524-4db8-b014-373cfeaf9fbc" (UID: "053a09bd-1524-4db8-b014-373cfeaf9fbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:39:28.141572 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.141472 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/053a09bd-1524-4db8-b014-373cfeaf9fbc-istio-token" (OuterVolumeSpecName: "istio-token") pod "053a09bd-1524-4db8-b014-373cfeaf9fbc" (UID: "053a09bd-1524-4db8-b014-373cfeaf9fbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:39:28.141647 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.141630 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/053a09bd-1524-4db8-b014-373cfeaf9fbc-kube-api-access-sjjhh" (OuterVolumeSpecName: "kube-api-access-sjjhh") pod "053a09bd-1524-4db8-b014-373cfeaf9fbc" (UID: "053a09bd-1524-4db8-b014-373cfeaf9fbc"). InnerVolumeSpecName "kube-api-access-sjjhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:39:28.141687 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.141669 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/053a09bd-1524-4db8-b014-373cfeaf9fbc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "053a09bd-1524-4db8-b014-373cfeaf9fbc" (UID: "053a09bd-1524-4db8-b014-373cfeaf9fbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 05 06:39:28.241044 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.240589 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjjhh\" (UniqueName: \"kubernetes.io/projected/053a09bd-1524-4db8-b014-373cfeaf9fbc-kube-api-access-sjjhh\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:39:28.241044 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.240628 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/053a09bd-1524-4db8-b014-373cfeaf9fbc-istio-envoy\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:39:28.241044 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.240643 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/053a09bd-1524-4db8-b014-373cfeaf9fbc-istio-token\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:39:28.241044 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:28.240656 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/053a09bd-1524-4db8-b014-373cfeaf9fbc-istio-podinfo\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:39:29.010338 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:29.010291 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh" Sep 05 06:39:29.049550 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:29.049511 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh"] Sep 05 06:39:29.056840 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:29.056801 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b1d1c87d-openshift-default-9d865f48d-jsbfh"] Sep 05 06:39:30.165849 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:30.165814 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="053a09bd-1524-4db8-b014-373cfeaf9fbc" path="/var/lib/kubelet/pods/053a09bd-1524-4db8-b014-373cfeaf9fbc/volumes" Sep 05 06:39:33.502788 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:33.502746 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r"] Sep 05 06:39:33.503540 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:33.503514 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="053a09bd-1524-4db8-b014-373cfeaf9fbc" containerName="istio-proxy" Sep 05 06:39:33.503540 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:33.503542 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="053a09bd-1524-4db8-b014-373cfeaf9fbc" containerName="istio-proxy" Sep 05 06:39:33.503713 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:33.503558 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="053a09bd-1524-4db8-b014-373cfeaf9fbc" containerName="istio-proxy" Sep 05 06:39:33.503713 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:33.503566 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="053a09bd-1524-4db8-b014-373cfeaf9fbc" containerName="istio-proxy" Sep 05 06:39:33.503713 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:33.503665 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="053a09bd-1524-4db8-b014-373cfeaf9fbc" containerName="istio-proxy" Sep 05 06:39:33.503713 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:33.503675 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="053a09bd-1524-4db8-b014-373cfeaf9fbc" containerName="istio-proxy" Sep 05 06:39:33.508487 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:33.508468 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r" Sep 05 06:39:33.511256 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:33.511221 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-2da41f75-dockercfg-cmnrk\"" Sep 05 06:39:33.511529 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:33.511505 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-2da41f75\"" Sep 05 06:39:33.516852 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:33.516827 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r"] Sep 05 06:39:33.582935 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:33.582899 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtwb9\" (UniqueName: \"kubernetes.io/projected/c1547044-bde4-420a-8688-8d9346e4f2ed-kube-api-access-qtwb9\") pod \"payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r\" (UID: \"c1547044-bde4-420a-8688-8d9346e4f2ed\") " pod="openshift-ingress/payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r" Sep 05 06:39:33.583131 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:33.582959 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1547044-bde4-420a-8688-8d9346e4f2ed-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r\" (UID: \"c1547044-bde4-420a-8688-8d9346e4f2ed\") " pod="openshift-ingress/payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r" Sep 05 06:39:33.683713 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:33.683671 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtwb9\" (UniqueName: \"kubernetes.io/projected/c1547044-bde4-420a-8688-8d9346e4f2ed-kube-api-access-qtwb9\") pod \"payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r\" (UID: \"c1547044-bde4-420a-8688-8d9346e4f2ed\") " pod="openshift-ingress/payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r" Sep 05 06:39:33.683894 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:33.683731 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1547044-bde4-420a-8688-8d9346e4f2ed-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r\" (UID: \"c1547044-bde4-420a-8688-8d9346e4f2ed\") " pod="openshift-ingress/payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r" Sep 05 06:39:33.684467 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:33.684451 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1547044-bde4-420a-8688-8d9346e4f2ed-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r\" (UID: \"c1547044-bde4-420a-8688-8d9346e4f2ed\") " pod="openshift-ingress/payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r" Sep 05 06:39:33.693864 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:33.693838 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtwb9\" (UniqueName: \"kubernetes.io/projected/c1547044-bde4-420a-8688-8d9346e4f2ed-kube-api-access-qtwb9\") pod \"payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r\" (UID: \"c1547044-bde4-420a-8688-8d9346e4f2ed\") " pod="openshift-ingress/payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r" Sep 05 06:39:33.821525 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:33.821438 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r" Sep 05 06:39:34.102225 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:34.102190 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r"] Sep 05 06:39:34.105513 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:39:34.105485 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1547044_bde4_420a_8688_8d9346e4f2ed.slice/crio-42191ba6ed4a4dfd5f69a68e21407d2d867432a3cb80bb2ad85f020ab78f5d04 WatchSource:0}: Error finding container 42191ba6ed4a4dfd5f69a68e21407d2d867432a3cb80bb2ad85f020ab78f5d04: Status 404 returned error can't find the container with id 42191ba6ed4a4dfd5f69a68e21407d2d867432a3cb80bb2ad85f020ab78f5d04 Sep 05 06:39:35.043873 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:35.043835 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r" event={"ID":"c1547044-bde4-420a-8688-8d9346e4f2ed","Type":"ContainerStarted","Data":"e7654ff64a263ca647eac9b9f0ce1198befca3d80b09345005010e198f755924"} Sep 05 06:39:35.044379 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:35.043891 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r" event={"ID":"c1547044-bde4-420a-8688-8d9346e4f2ed","Type":"ContainerStarted","Data":"42191ba6ed4a4dfd5f69a68e21407d2d867432a3cb80bb2ad85f020ab78f5d04"} Sep 05 06:39:35.044379 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:35.043922 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r" Sep 05 06:39:35.064774 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:35.064720 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r" podStartSLOduration=2.064701703 podStartE2EDuration="2.064701703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:39:35.061611431 +0000 UTC m=+799.446154563" watchObservedRunningTime="2026-09-05 06:39:35.064701703 +0000 UTC m=+799.449244826" Sep 05 06:39:46.051885 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:39:46.051805 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r" Sep 05 06:40:04.202734 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:04.202675 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r"] Sep 05 06:40:04.203425 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:04.202968 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r" podUID="c1547044-bde4-420a-8688-8d9346e4f2ed" containerName="payload-processing" containerID="cri-o://e7654ff64a263ca647eac9b9f0ce1198befca3d80b09345005010e198f755924" gracePeriod=30 Sep 05 06:40:04.471384 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:04.471355 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r" Sep 05 06:40:04.512606 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:04.510886 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1547044-bde4-420a-8688-8d9346e4f2ed-plugins-config-volume\") pod \"c1547044-bde4-420a-8688-8d9346e4f2ed\" (UID: \"c1547044-bde4-420a-8688-8d9346e4f2ed\") " Sep 05 06:40:04.512606 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:04.511037 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtwb9\" (UniqueName: \"kubernetes.io/projected/c1547044-bde4-420a-8688-8d9346e4f2ed-kube-api-access-qtwb9\") pod \"c1547044-bde4-420a-8688-8d9346e4f2ed\" (UID: \"c1547044-bde4-420a-8688-8d9346e4f2ed\") " Sep 05 06:40:04.518096 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:04.518051 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1547044-bde4-420a-8688-8d9346e4f2ed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c1547044-bde4-420a-8688-8d9346e4f2ed" (UID: "c1547044-bde4-420a-8688-8d9346e4f2ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:40:04.519420 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:04.519384 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1547044-bde4-420a-8688-8d9346e4f2ed-kube-api-access-qtwb9" (OuterVolumeSpecName: "kube-api-access-qtwb9") pod "c1547044-bde4-420a-8688-8d9346e4f2ed" (UID: "c1547044-bde4-420a-8688-8d9346e4f2ed"). InnerVolumeSpecName "kube-api-access-qtwb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:40:04.612455 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:04.612417 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtwb9\" (UniqueName: \"kubernetes.io/projected/c1547044-bde4-420a-8688-8d9346e4f2ed-kube-api-access-qtwb9\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:40:04.612455 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:04.612447 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1547044-bde4-420a-8688-8d9346e4f2ed-plugins-config-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:40:05.201375 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:05.201337 2565 generic.go:356] "Generic (PLEG): container finished" podID="c1547044-bde4-420a-8688-8d9346e4f2ed" containerID="e7654ff64a263ca647eac9b9f0ce1198befca3d80b09345005010e198f755924" exitCode=0 Sep 05 06:40:05.201587 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:05.201386 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r" Sep 05 06:40:05.201587 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:05.201403 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r" event={"ID":"c1547044-bde4-420a-8688-8d9346e4f2ed","Type":"ContainerDied","Data":"e7654ff64a263ca647eac9b9f0ce1198befca3d80b09345005010e198f755924"} Sep 05 06:40:05.201587 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:05.201471 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r" event={"ID":"c1547044-bde4-420a-8688-8d9346e4f2ed","Type":"ContainerDied","Data":"42191ba6ed4a4dfd5f69a68e21407d2d867432a3cb80bb2ad85f020ab78f5d04"} Sep 05 06:40:05.201587 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:05.201500 2565 scope.go:117] "RemoveContainer" containerID="e7654ff64a263ca647eac9b9f0ce1198befca3d80b09345005010e198f755924" Sep 05 06:40:05.213278 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:05.213006 2565 scope.go:117] "RemoveContainer" containerID="e7654ff64a263ca647eac9b9f0ce1198befca3d80b09345005010e198f755924" Sep 05 06:40:05.213631 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:40:05.213528 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7654ff64a263ca647eac9b9f0ce1198befca3d80b09345005010e198f755924\": container with ID starting with e7654ff64a263ca647eac9b9f0ce1198befca3d80b09345005010e198f755924 not found: ID does not exist" containerID="e7654ff64a263ca647eac9b9f0ce1198befca3d80b09345005010e198f755924" Sep 05 06:40:05.213631 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:05.213558 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7654ff64a263ca647eac9b9f0ce1198befca3d80b09345005010e198f755924"} err="failed to get container status \"e7654ff64a263ca647eac9b9f0ce1198befca3d80b09345005010e198f755924\": rpc error: code = NotFound desc = could not find container \"e7654ff64a263ca647eac9b9f0ce1198befca3d80b09345005010e198f755924\": container with ID starting with e7654ff64a263ca647eac9b9f0ce1198befca3d80b09345005010e198f755924 not found: ID does not exist" Sep 05 06:40:05.226088 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:05.226054 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r"] Sep 05 06:40:05.231322 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:05.231272 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2da41f75-785967b74-vvp9r"] Sep 05 06:40:06.166737 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:06.166702 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1547044-bde4-420a-8688-8d9346e4f2ed" path="/var/lib/kubelet/pods/c1547044-bde4-420a-8688-8d9346e4f2ed/volumes" Sep 05 06:40:09.800751 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:09.797716 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk"] Sep 05 06:40:09.800751 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:09.798629 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1547044-bde4-420a-8688-8d9346e4f2ed" containerName="payload-processing" Sep 05 06:40:09.800751 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:09.798651 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1547044-bde4-420a-8688-8d9346e4f2ed" containerName="payload-processing" Sep 05 06:40:09.800751 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:09.798862 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1547044-bde4-420a-8688-8d9346e4f2ed" containerName="payload-processing" Sep 05 06:40:09.804666 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:09.804630 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk" Sep 05 06:40:09.807453 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:09.807424 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-bba7714e\"" Sep 05 06:40:09.807617 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:09.807518 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-bba7714e-dockercfg-8zbbd\"" Sep 05 06:40:09.814938 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:09.814915 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk"] Sep 05 06:40:09.864027 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:09.863987 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f26ad8fa-40ae-4a17-bd70-08c73f73beab-plugins-config-volume\") pod \"payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk\" (UID: \"f26ad8fa-40ae-4a17-bd70-08c73f73beab\") " pod="openshift-ingress/payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk" Sep 05 06:40:09.864495 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:09.864467 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cmgg\" (UniqueName: \"kubernetes.io/projected/f26ad8fa-40ae-4a17-bd70-08c73f73beab-kube-api-access-7cmgg\") pod \"payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk\" (UID: \"f26ad8fa-40ae-4a17-bd70-08c73f73beab\") " pod="openshift-ingress/payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk" Sep 05 06:40:09.965425 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:09.965368 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmgg\" (UniqueName: \"kubernetes.io/projected/f26ad8fa-40ae-4a17-bd70-08c73f73beab-kube-api-access-7cmgg\") pod \"payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk\" (UID: \"f26ad8fa-40ae-4a17-bd70-08c73f73beab\") " pod="openshift-ingress/payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk" Sep 05 06:40:09.965630 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:09.965455 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f26ad8fa-40ae-4a17-bd70-08c73f73beab-plugins-config-volume\") pod \"payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk\" (UID: \"f26ad8fa-40ae-4a17-bd70-08c73f73beab\") " pod="openshift-ingress/payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk" Sep 05 06:40:09.966131 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:09.966111 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f26ad8fa-40ae-4a17-bd70-08c73f73beab-plugins-config-volume\") pod \"payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk\" (UID: \"f26ad8fa-40ae-4a17-bd70-08c73f73beab\") " pod="openshift-ingress/payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk" Sep 05 06:40:09.977578 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:09.977540 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cmgg\" (UniqueName: \"kubernetes.io/projected/f26ad8fa-40ae-4a17-bd70-08c73f73beab-kube-api-access-7cmgg\") pod \"payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk\" (UID: \"f26ad8fa-40ae-4a17-bd70-08c73f73beab\") " pod="openshift-ingress/payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk" Sep 05 06:40:10.118028 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:10.117992 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk" Sep 05 06:40:10.394668 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:10.394642 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk"] Sep 05 06:40:10.397320 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:40:10.397263 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26ad8fa_40ae_4a17_bd70_08c73f73beab.slice/crio-50b191cef45e2f605eea5ba0eb197fbb6cfccf6bf1eff8d92425054cb62ba288 WatchSource:0}: Error finding container 50b191cef45e2f605eea5ba0eb197fbb6cfccf6bf1eff8d92425054cb62ba288: Status 404 returned error can't find the container with id 50b191cef45e2f605eea5ba0eb197fbb6cfccf6bf1eff8d92425054cb62ba288 Sep 05 06:40:11.235820 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:11.235785 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk" event={"ID":"f26ad8fa-40ae-4a17-bd70-08c73f73beab","Type":"ContainerStarted","Data":"62d75b94041e6f774c1f0169d0ebdcaaf2d88a4338d8ab54ca18dbc3fd243174"} Sep 05 06:40:11.236288 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:11.235843 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk" event={"ID":"f26ad8fa-40ae-4a17-bd70-08c73f73beab","Type":"ContainerStarted","Data":"50b191cef45e2f605eea5ba0eb197fbb6cfccf6bf1eff8d92425054cb62ba288"} Sep 05 06:40:11.236288 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:11.235954 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk" Sep 05 06:40:11.261139 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:11.261038 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk" podStartSLOduration=2.261003816 podStartE2EDuration="2.261003816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:40:11.260832396 +0000 UTC m=+835.645375545" watchObservedRunningTime="2026-09-05 06:40:11.261003816 +0000 UTC m=+835.645546939" Sep 05 06:40:22.243664 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:22.243626 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk" Sep 05 06:40:33.552888 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:33.552851 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bba7714e-w9rjv"] Sep 05 06:40:33.558008 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:33.557986 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bba7714e-w9rjv" Sep 05 06:40:33.560945 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:33.560921 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slfqj\"" Sep 05 06:40:33.561130 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:33.560933 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 05 06:40:33.565427 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:33.565402 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bba7714e-w9rjv"] Sep 05 06:40:33.621190 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:33.621157 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cff11cd9-1d4b-4385-b5eb-3785ee41af52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bba7714e-w9rjv\" (UID: \"cff11cd9-1d4b-4385-b5eb-3785ee41af52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bba7714e-w9rjv" Sep 05 06:40:33.722371 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:33.722300 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cff11cd9-1d4b-4385-b5eb-3785ee41af52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bba7714e-w9rjv\" (UID: \"cff11cd9-1d4b-4385-b5eb-3785ee41af52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bba7714e-w9rjv" Sep 05 06:40:33.722965 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:33.722947 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cff11cd9-1d4b-4385-b5eb-3785ee41af52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bba7714e-w9rjv\" (UID: \"cff11cd9-1d4b-4385-b5eb-3785ee41af52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bba7714e-w9rjv" Sep 05 06:40:33.870691 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:33.870651 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bba7714e-w9rjv" Sep 05 06:40:34.143963 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:34.143936 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bba7714e-w9rjv"] Sep 05 06:40:34.146589 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:40:34.146559 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff11cd9_1d4b_4385_b5eb_3785ee41af52.slice/crio-0b8761a51b0d6c2b6743dc6a17b8d5f2f1ee1b95527dbab8b0da26e1a69a6872 WatchSource:0}: Error finding container 0b8761a51b0d6c2b6743dc6a17b8d5f2f1ee1b95527dbab8b0da26e1a69a6872: Status 404 returned error can't find the container with id 0b8761a51b0d6c2b6743dc6a17b8d5f2f1ee1b95527dbab8b0da26e1a69a6872 Sep 05 06:40:34.148517 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:34.148501 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 05 06:40:34.363027 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:34.362993 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bba7714e-w9rjv" event={"ID":"cff11cd9-1d4b-4385-b5eb-3785ee41af52","Type":"ContainerStarted","Data":"0b8761a51b0d6c2b6743dc6a17b8d5f2f1ee1b95527dbab8b0da26e1a69a6872"} Sep 05 06:40:37.380016 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:37.379984 2565 generic.go:356] "Generic (PLEG): container finished" podID="cff11cd9-1d4b-4385-b5eb-3785ee41af52" containerID="54e73981fb191f9d4bac05af806bb7aa107805d48d0435d159f0678ff735f052" exitCode=0 Sep 05 06:40:37.380494 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:37.380076 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bba7714e-w9rjv" event={"ID":"cff11cd9-1d4b-4385-b5eb-3785ee41af52","Type":"ContainerDied","Data":"54e73981fb191f9d4bac05af806bb7aa107805d48d0435d159f0678ff735f052"} Sep 05 06:40:38.523827 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:38.523800 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bba7714e-w9rjv" Sep 05 06:40:38.570255 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:38.570225 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cff11cd9-1d4b-4385-b5eb-3785ee41af52-maas-api-service-ca\") pod \"cff11cd9-1d4b-4385-b5eb-3785ee41af52\" (UID: \"cff11cd9-1d4b-4385-b5eb-3785ee41af52\") " Sep 05 06:40:38.570619 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:38.570598 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cff11cd9-1d4b-4385-b5eb-3785ee41af52-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cff11cd9-1d4b-4385-b5eb-3785ee41af52" (UID: "cff11cd9-1d4b-4385-b5eb-3785ee41af52"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:40:38.671802 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:38.671698 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cff11cd9-1d4b-4385-b5eb-3785ee41af52-maas-api-service-ca\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:40:39.392536 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:39.392502 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bba7714e-w9rjv" Sep 05 06:40:39.392727 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:39.392502 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bba7714e-w9rjv" event={"ID":"cff11cd9-1d4b-4385-b5eb-3785ee41af52","Type":"ContainerDied","Data":"0b8761a51b0d6c2b6743dc6a17b8d5f2f1ee1b95527dbab8b0da26e1a69a6872"} Sep 05 06:40:39.392727 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:39.392646 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b8761a51b0d6c2b6743dc6a17b8d5f2f1ee1b95527dbab8b0da26e1a69a6872" Sep 05 06:40:41.967877 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:41.967841 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6"] Sep 05 06:40:41.968523 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:41.968489 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6" podUID="cc9bfab8-27f1-489b-927e-fcbc95b482c0" containerName="payload-pre-processing" containerID="cri-o://a0abbb94b64b3e7f637cf0663dd96f7ce80274d4a321756c88e4e86271740b35" gracePeriod=30 Sep 05 06:40:42.256731 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:42.256432 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6" Sep 05 06:40:42.311578 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:42.311550 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp4l7\" (UniqueName: \"kubernetes.io/projected/cc9bfab8-27f1-489b-927e-fcbc95b482c0-kube-api-access-mp4l7\") pod \"cc9bfab8-27f1-489b-927e-fcbc95b482c0\" (UID: \"cc9bfab8-27f1-489b-927e-fcbc95b482c0\") " Sep 05 06:40:42.311778 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:42.311619 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9bfab8-27f1-489b-927e-fcbc95b482c0-plugins-config-volume\") pod \"cc9bfab8-27f1-489b-927e-fcbc95b482c0\" (UID: \"cc9bfab8-27f1-489b-927e-fcbc95b482c0\") " Sep 05 06:40:42.312008 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:42.311981 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9bfab8-27f1-489b-927e-fcbc95b482c0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cc9bfab8-27f1-489b-927e-fcbc95b482c0" (UID: "cc9bfab8-27f1-489b-927e-fcbc95b482c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:40:42.313855 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:42.313828 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9bfab8-27f1-489b-927e-fcbc95b482c0-kube-api-access-mp4l7" (OuterVolumeSpecName: "kube-api-access-mp4l7") pod "cc9bfab8-27f1-489b-927e-fcbc95b482c0" (UID: "cc9bfab8-27f1-489b-927e-fcbc95b482c0"). InnerVolumeSpecName "kube-api-access-mp4l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:40:42.412376 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:42.412345 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp4l7\" (UniqueName: \"kubernetes.io/projected/cc9bfab8-27f1-489b-927e-fcbc95b482c0-kube-api-access-mp4l7\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:40:42.412376 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:42.412367 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9bfab8-27f1-489b-927e-fcbc95b482c0-plugins-config-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:40:42.412988 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:42.412968 2565 generic.go:356] "Generic (PLEG): container finished" podID="cc9bfab8-27f1-489b-927e-fcbc95b482c0" containerID="a0abbb94b64b3e7f637cf0663dd96f7ce80274d4a321756c88e4e86271740b35" exitCode=0 Sep 05 06:40:42.413060 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:42.413002 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6" event={"ID":"cc9bfab8-27f1-489b-927e-fcbc95b482c0","Type":"ContainerDied","Data":"a0abbb94b64b3e7f637cf0663dd96f7ce80274d4a321756c88e4e86271740b35"} Sep 05 06:40:42.413060 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:42.413022 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6" Sep 05 06:40:42.413060 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:42.413039 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6" event={"ID":"cc9bfab8-27f1-489b-927e-fcbc95b482c0","Type":"ContainerDied","Data":"1b1c1be6a167a0b56028d8138b71d9e30394bf82d2f1dc602e15e41f7a1e1e24"} Sep 05 06:40:42.413157 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:42.413061 2565 scope.go:117] "RemoveContainer" containerID="a0abbb94b64b3e7f637cf0663dd96f7ce80274d4a321756c88e4e86271740b35" Sep 05 06:40:42.422383 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:42.422362 2565 scope.go:117] "RemoveContainer" containerID="a0abbb94b64b3e7f637cf0663dd96f7ce80274d4a321756c88e4e86271740b35" Sep 05 06:40:42.422661 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:40:42.422640 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0abbb94b64b3e7f637cf0663dd96f7ce80274d4a321756c88e4e86271740b35\": container with ID starting with a0abbb94b64b3e7f637cf0663dd96f7ce80274d4a321756c88e4e86271740b35 not found: ID does not exist" containerID="a0abbb94b64b3e7f637cf0663dd96f7ce80274d4a321756c88e4e86271740b35" Sep 05 06:40:42.422726 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:42.422669 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0abbb94b64b3e7f637cf0663dd96f7ce80274d4a321756c88e4e86271740b35"} err="failed to get container status \"a0abbb94b64b3e7f637cf0663dd96f7ce80274d4a321756c88e4e86271740b35\": rpc error: code = NotFound desc = could not find container \"a0abbb94b64b3e7f637cf0663dd96f7ce80274d4a321756c88e4e86271740b35\": container with ID starting with a0abbb94b64b3e7f637cf0663dd96f7ce80274d4a321756c88e4e86271740b35 not found: ID does not exist" Sep 05 06:40:42.439668 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:42.439643 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6"] Sep 05 06:40:42.441526 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:42.441507 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8956b5-7cc79879d6-89ff6"] Sep 05 06:40:44.165675 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:44.165638 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc9bfab8-27f1-489b-927e-fcbc95b482c0" path="/var/lib/kubelet/pods/cc9bfab8-27f1-489b-927e-fcbc95b482c0/volumes" Sep 05 06:40:49.281418 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:49.281372 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk"] Sep 05 06:40:49.281934 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:49.281639 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" podUID="da2ee56d-6635-4935-8dd0-ead921efff47" containerName="istio-proxy" containerID="cri-o://9d21ce782c25220f14d0ae53edeb07a9fee46104da1246e3a666e5b9470db879" gracePeriod=30 Sep 05 06:40:49.964814 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:49.964781 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk"] Sep 05 06:40:49.965071 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:49.965044 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk" podUID="f26ad8fa-40ae-4a17-bd70-08c73f73beab" containerName="payload-processing" containerID="cri-o://62d75b94041e6f774c1f0169d0ebdcaaf2d88a4338d8ab54ca18dbc3fd243174" gracePeriod=30 Sep 05 06:40:50.203640 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:50.203605 2565 patch_prober.go:28] interesting pod/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 05 06:40:50.203791 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:50.203666 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" podUID="da2ee56d-6635-4935-8dd0-ead921efff47" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 05 06:40:50.228530 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:50.228469 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk" Sep 05 06:40:50.293024 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:50.292989 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f26ad8fa-40ae-4a17-bd70-08c73f73beab-plugins-config-volume\") pod \"f26ad8fa-40ae-4a17-bd70-08c73f73beab\" (UID: \"f26ad8fa-40ae-4a17-bd70-08c73f73beab\") " Sep 05 06:40:50.293448 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:50.293113 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cmgg\" (UniqueName: \"kubernetes.io/projected/f26ad8fa-40ae-4a17-bd70-08c73f73beab-kube-api-access-7cmgg\") pod \"f26ad8fa-40ae-4a17-bd70-08c73f73beab\" (UID: \"f26ad8fa-40ae-4a17-bd70-08c73f73beab\") " Sep 05 06:40:50.293448 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:50.293374 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f26ad8fa-40ae-4a17-bd70-08c73f73beab-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f26ad8fa-40ae-4a17-bd70-08c73f73beab" (UID: "f26ad8fa-40ae-4a17-bd70-08c73f73beab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:40:50.293613 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:50.293596 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f26ad8fa-40ae-4a17-bd70-08c73f73beab-plugins-config-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:40:50.295302 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:50.295282 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bba7714e-w9rjv"] Sep 05 06:40:50.295403 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:50.295381 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f26ad8fa-40ae-4a17-bd70-08c73f73beab-kube-api-access-7cmgg" (OuterVolumeSpecName: "kube-api-access-7cmgg") pod "f26ad8fa-40ae-4a17-bd70-08c73f73beab" (UID: "f26ad8fa-40ae-4a17-bd70-08c73f73beab"). InnerVolumeSpecName "kube-api-access-7cmgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:40:50.304178 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:50.304151 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bba7714e-w9rjv"] Sep 05 06:40:50.394369 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:50.394293 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cmgg\" (UniqueName: \"kubernetes.io/projected/f26ad8fa-40ae-4a17-bd70-08c73f73beab-kube-api-access-7cmgg\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:40:50.455495 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:50.455462 2565 generic.go:356] "Generic (PLEG): container finished" podID="f26ad8fa-40ae-4a17-bd70-08c73f73beab" containerID="62d75b94041e6f774c1f0169d0ebdcaaf2d88a4338d8ab54ca18dbc3fd243174" exitCode=0 Sep 05 06:40:50.455666 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:50.455508 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk" Sep 05 06:40:50.455666 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:50.455515 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk" event={"ID":"f26ad8fa-40ae-4a17-bd70-08c73f73beab","Type":"ContainerDied","Data":"62d75b94041e6f774c1f0169d0ebdcaaf2d88a4338d8ab54ca18dbc3fd243174"} Sep 05 06:40:50.455666 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:50.455565 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk" event={"ID":"f26ad8fa-40ae-4a17-bd70-08c73f73beab","Type":"ContainerDied","Data":"50b191cef45e2f605eea5ba0eb197fbb6cfccf6bf1eff8d92425054cb62ba288"} Sep 05 06:40:50.455666 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:50.455588 2565 scope.go:117] "RemoveContainer" containerID="62d75b94041e6f774c1f0169d0ebdcaaf2d88a4338d8ab54ca18dbc3fd243174" Sep 05 06:40:50.465623 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:50.465599 2565 scope.go:117] "RemoveContainer" containerID="62d75b94041e6f774c1f0169d0ebdcaaf2d88a4338d8ab54ca18dbc3fd243174" Sep 05 06:40:50.465959 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:40:50.465938 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62d75b94041e6f774c1f0169d0ebdcaaf2d88a4338d8ab54ca18dbc3fd243174\": container with ID starting with 62d75b94041e6f774c1f0169d0ebdcaaf2d88a4338d8ab54ca18dbc3fd243174 not found: ID does not exist" containerID="62d75b94041e6f774c1f0169d0ebdcaaf2d88a4338d8ab54ca18dbc3fd243174" Sep 05 06:40:50.466051 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:50.465967 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d75b94041e6f774c1f0169d0ebdcaaf2d88a4338d8ab54ca18dbc3fd243174"} err="failed to get container status \"62d75b94041e6f774c1f0169d0ebdcaaf2d88a4338d8ab54ca18dbc3fd243174\": rpc error: code = NotFound desc = could not find container \"62d75b94041e6f774c1f0169d0ebdcaaf2d88a4338d8ab54ca18dbc3fd243174\": container with ID starting with 62d75b94041e6f774c1f0169d0ebdcaaf2d88a4338d8ab54ca18dbc3fd243174 not found: ID does not exist" Sep 05 06:40:50.487604 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:50.487535 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk"] Sep 05 06:40:50.493456 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:50.493430 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bba7714e-8575dcbdb5-pcmzk"] Sep 05 06:40:52.169476 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:52.169439 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cff11cd9-1d4b-4385-b5eb-3785ee41af52" path="/var/lib/kubelet/pods/cff11cd9-1d4b-4385-b5eb-3785ee41af52/volumes" Sep 05 06:40:52.169919 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:52.169818 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f26ad8fa-40ae-4a17-bd70-08c73f73beab" path="/var/lib/kubelet/pods/f26ad8fa-40ae-4a17-bd70-08c73f73beab/volumes" Sep 05 06:40:54.486124 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.486086 2565 generic.go:356] "Generic (PLEG): container finished" podID="da2ee56d-6635-4935-8dd0-ead921efff47" containerID="9d21ce782c25220f14d0ae53edeb07a9fee46104da1246e3a666e5b9470db879" exitCode=0 Sep 05 06:40:54.486608 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.486162 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" event={"ID":"da2ee56d-6635-4935-8dd0-ead921efff47","Type":"ContainerDied","Data":"9d21ce782c25220f14d0ae53edeb07a9fee46104da1246e3a666e5b9470db879"} Sep 05 06:40:54.486608 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.486243 2565 scope.go:117] "RemoveContainer" containerID="9ddca6f1b58df477d5840b409012f2a40c9cfc0a00820590228d6db5f467e193" Sep 05 06:40:54.547714 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.547658 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:40:54.636328 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.636279 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-istio-envoy\") pod \"da2ee56d-6635-4935-8dd0-ead921efff47\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " Sep 05 06:40:54.636478 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.636348 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da2ee56d-6635-4935-8dd0-ead921efff47-istiod-ca-cert\") pod \"da2ee56d-6635-4935-8dd0-ead921efff47\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " Sep 05 06:40:54.636478 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.636392 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-workload-socket\") pod \"da2ee56d-6635-4935-8dd0-ead921efff47\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " Sep 05 06:40:54.636478 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.636427 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da2ee56d-6635-4935-8dd0-ead921efff47-istio-podinfo\") pod \"da2ee56d-6635-4935-8dd0-ead921efff47\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " Sep 05 06:40:54.636478 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.636467 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdw6m\" (UniqueName: \"kubernetes.io/projected/da2ee56d-6635-4935-8dd0-ead921efff47-kube-api-access-gdw6m\") pod \"da2ee56d-6635-4935-8dd0-ead921efff47\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " Sep 05 06:40:54.636704 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.636484 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-credential-socket\") pod \"da2ee56d-6635-4935-8dd0-ead921efff47\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " Sep 05 06:40:54.636704 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.636606 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-istio-data\") pod \"da2ee56d-6635-4935-8dd0-ead921efff47\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " Sep 05 06:40:54.636704 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.636636 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-workload-certs\") pod \"da2ee56d-6635-4935-8dd0-ead921efff47\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " Sep 05 06:40:54.636704 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.636665 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da2ee56d-6635-4935-8dd0-ead921efff47-istio-token\") pod \"da2ee56d-6635-4935-8dd0-ead921efff47\" (UID: \"da2ee56d-6635-4935-8dd0-ead921efff47\") " Sep 05 06:40:54.636913 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.636692 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "da2ee56d-6635-4935-8dd0-ead921efff47" (UID: "da2ee56d-6635-4935-8dd0-ead921efff47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:40:54.636913 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.636715 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da2ee56d-6635-4935-8dd0-ead921efff47-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "da2ee56d-6635-4935-8dd0-ead921efff47" (UID: "da2ee56d-6635-4935-8dd0-ead921efff47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:40:54.636913 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.636884 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-istio-data" (OuterVolumeSpecName: "istio-data") pod "da2ee56d-6635-4935-8dd0-ead921efff47" (UID: "da2ee56d-6635-4935-8dd0-ead921efff47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:40:54.637290 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.636933 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "da2ee56d-6635-4935-8dd0-ead921efff47" (UID: "da2ee56d-6635-4935-8dd0-ead921efff47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:40:54.637290 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.637056 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-istio-data\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:40:54.637290 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.637068 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-workload-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:40:54.637290 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.637077 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da2ee56d-6635-4935-8dd0-ead921efff47-istiod-ca-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:40:54.637290 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.637085 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-workload-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:40:54.637290 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.637077 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "da2ee56d-6635-4935-8dd0-ead921efff47" (UID: "da2ee56d-6635-4935-8dd0-ead921efff47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:40:54.638830 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.638805 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "da2ee56d-6635-4935-8dd0-ead921efff47" (UID: "da2ee56d-6635-4935-8dd0-ead921efff47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:40:54.638830 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.638807 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da2ee56d-6635-4935-8dd0-ead921efff47-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "da2ee56d-6635-4935-8dd0-ead921efff47" (UID: "da2ee56d-6635-4935-8dd0-ead921efff47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 05 06:40:54.639030 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.638927 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da2ee56d-6635-4935-8dd0-ead921efff47-istio-token" (OuterVolumeSpecName: "istio-token") pod "da2ee56d-6635-4935-8dd0-ead921efff47" (UID: "da2ee56d-6635-4935-8dd0-ead921efff47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:40:54.639030 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.638943 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da2ee56d-6635-4935-8dd0-ead921efff47-kube-api-access-gdw6m" (OuterVolumeSpecName: "kube-api-access-gdw6m") pod "da2ee56d-6635-4935-8dd0-ead921efff47" (UID: "da2ee56d-6635-4935-8dd0-ead921efff47"). InnerVolumeSpecName "kube-api-access-gdw6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:40:54.738085 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.738044 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdw6m\" (UniqueName: \"kubernetes.io/projected/da2ee56d-6635-4935-8dd0-ead921efff47-kube-api-access-gdw6m\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:40:54.738085 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.738074 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-credential-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:40:54.738085 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.738085 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da2ee56d-6635-4935-8dd0-ead921efff47-istio-token\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:40:54.738085 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.738094 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da2ee56d-6635-4935-8dd0-ead921efff47-istio-envoy\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:40:54.738386 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:54.738103 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da2ee56d-6635-4935-8dd0-ead921efff47-istio-podinfo\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:40:55.493029 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:55.492992 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" event={"ID":"da2ee56d-6635-4935-8dd0-ead921efff47","Type":"ContainerDied","Data":"ecac8d3bb284bcd80ed8e12715b0fc2c4427835f0c3f145289308582a1e87c44"} Sep 05 06:40:55.493029 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:55.493019 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk" Sep 05 06:40:55.493535 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:55.493050 2565 scope.go:117] "RemoveContainer" containerID="9d21ce782c25220f14d0ae53edeb07a9fee46104da1246e3a666e5b9470db879" Sep 05 06:40:55.518779 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:55.518750 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk"] Sep 05 06:40:55.524887 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:55.524863 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-8956b5-openshift-default-6df9b9ccb-qc7lk"] Sep 05 06:40:56.166995 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:56.166959 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da2ee56d-6635-4935-8dd0-ead921efff47" path="/var/lib/kubelet/pods/da2ee56d-6635-4935-8dd0-ead921efff47/volumes" Sep 05 06:40:57.409079 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.409041 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd"] Sep 05 06:40:57.409448 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.409440 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc9bfab8-27f1-489b-927e-fcbc95b482c0" containerName="payload-pre-processing" Sep 05 06:40:57.409508 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.409451 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc9bfab8-27f1-489b-927e-fcbc95b482c0" containerName="payload-pre-processing" Sep 05 06:40:57.409508 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.409462 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f26ad8fa-40ae-4a17-bd70-08c73f73beab" containerName="payload-processing" Sep 05 06:40:57.409508 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.409467 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f26ad8fa-40ae-4a17-bd70-08c73f73beab" containerName="payload-processing" Sep 05 06:40:57.409508 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.409478 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da2ee56d-6635-4935-8dd0-ead921efff47" containerName="istio-proxy" Sep 05 06:40:57.409508 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.409484 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="da2ee56d-6635-4935-8dd0-ead921efff47" containerName="istio-proxy" Sep 05 06:40:57.409508 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.409495 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da2ee56d-6635-4935-8dd0-ead921efff47" containerName="istio-proxy" Sep 05 06:40:57.409508 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.409500 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="da2ee56d-6635-4935-8dd0-ead921efff47" containerName="istio-proxy" Sep 05 06:40:57.409508 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.409505 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cff11cd9-1d4b-4385-b5eb-3785ee41af52" containerName="revoke-keys" Sep 05 06:40:57.409508 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.409510 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cff11cd9-1d4b-4385-b5eb-3785ee41af52" containerName="revoke-keys" Sep 05 06:40:57.409849 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.409588 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f26ad8fa-40ae-4a17-bd70-08c73f73beab" containerName="payload-processing" Sep 05 06:40:57.409849 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.409596 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cff11cd9-1d4b-4385-b5eb-3785ee41af52" containerName="revoke-keys" Sep 05 06:40:57.409849 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.409602 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="da2ee56d-6635-4935-8dd0-ead921efff47" containerName="istio-proxy" Sep 05 06:40:57.409849 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.409607 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc9bfab8-27f1-489b-927e-fcbc95b482c0" containerName="payload-pre-processing" Sep 05 06:40:57.409849 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.409614 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="da2ee56d-6635-4935-8dd0-ead921efff47" containerName="istio-proxy" Sep 05 06:40:57.415140 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.415115 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd" Sep 05 06:40:57.417728 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.417705 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-13503c3c\"" Sep 05 06:40:57.425706 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.425682 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd"] Sep 05 06:40:57.428322 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.428231 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw"] Sep 05 06:40:57.432649 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.432627 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw" Sep 05 06:40:57.435135 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.435113 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-13503c3c-dockercfg-h58f4\"" Sep 05 06:40:57.442856 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.442833 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw"] Sep 05 06:40:57.568964 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.568927 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47jwl\" (UniqueName: \"kubernetes.io/projected/2991e799-10c4-4bd8-9a2a-ec2a94350816-kube-api-access-47jwl\") pod \"payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw\" (UID: \"2991e799-10c4-4bd8-9a2a-ec2a94350816\") " pod="openshift-ingress/payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw" Sep 05 06:40:57.568964 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.568970 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52449020-e29a-4edd-9f20-d0a2ca28d380-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd\" (UID: \"52449020-e29a-4edd-9f20-d0a2ca28d380\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd" Sep 05 06:40:57.569188 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.569068 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwndf\" (UniqueName: \"kubernetes.io/projected/52449020-e29a-4edd-9f20-d0a2ca28d380-kube-api-access-dwndf\") pod \"payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd\" (UID: \"52449020-e29a-4edd-9f20-d0a2ca28d380\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd" Sep 05 06:40:57.569188 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.569134 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2991e799-10c4-4bd8-9a2a-ec2a94350816-plugins-config-volume\") pod \"payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw\" (UID: \"2991e799-10c4-4bd8-9a2a-ec2a94350816\") " pod="openshift-ingress/payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw" Sep 05 06:40:57.670282 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.670173 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwndf\" (UniqueName: \"kubernetes.io/projected/52449020-e29a-4edd-9f20-d0a2ca28d380-kube-api-access-dwndf\") pod \"payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd\" (UID: \"52449020-e29a-4edd-9f20-d0a2ca28d380\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd" Sep 05 06:40:57.670282 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.670259 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2991e799-10c4-4bd8-9a2a-ec2a94350816-plugins-config-volume\") pod \"payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw\" (UID: \"2991e799-10c4-4bd8-9a2a-ec2a94350816\") " pod="openshift-ingress/payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw" Sep 05 06:40:57.670565 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.670355 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47jwl\" (UniqueName: \"kubernetes.io/projected/2991e799-10c4-4bd8-9a2a-ec2a94350816-kube-api-access-47jwl\") pod \"payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw\" (UID: \"2991e799-10c4-4bd8-9a2a-ec2a94350816\") " pod="openshift-ingress/payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw" Sep 05 06:40:57.670565 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.670382 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52449020-e29a-4edd-9f20-d0a2ca28d380-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd\" (UID: \"52449020-e29a-4edd-9f20-d0a2ca28d380\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd" Sep 05 06:40:57.671105 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.671079 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52449020-e29a-4edd-9f20-d0a2ca28d380-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd\" (UID: \"52449020-e29a-4edd-9f20-d0a2ca28d380\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd" Sep 05 06:40:57.671222 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.671198 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2991e799-10c4-4bd8-9a2a-ec2a94350816-plugins-config-volume\") pod \"payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw\" (UID: \"2991e799-10c4-4bd8-9a2a-ec2a94350816\") " pod="openshift-ingress/payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw" Sep 05 06:40:57.681959 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.681929 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwndf\" (UniqueName: \"kubernetes.io/projected/52449020-e29a-4edd-9f20-d0a2ca28d380-kube-api-access-dwndf\") pod \"payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd\" (UID: \"52449020-e29a-4edd-9f20-d0a2ca28d380\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd" Sep 05 06:40:57.682078 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.682057 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47jwl\" (UniqueName: \"kubernetes.io/projected/2991e799-10c4-4bd8-9a2a-ec2a94350816-kube-api-access-47jwl\") pod \"payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw\" (UID: \"2991e799-10c4-4bd8-9a2a-ec2a94350816\") " pod="openshift-ingress/payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw" Sep 05 06:40:57.727608 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.727569 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd" Sep 05 06:40:57.745638 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:57.745610 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw" Sep 05 06:40:58.020850 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:58.020816 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd"] Sep 05 06:40:58.022418 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:40:58.022388 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52449020_e29a_4edd_9f20_d0a2ca28d380.slice/crio-e40b439837fe1d91457b99d2a10d89fc3576a663d611a4130e1cd75fbc627db8 WatchSource:0}: Error finding container e40b439837fe1d91457b99d2a10d89fc3576a663d611a4130e1cd75fbc627db8: Status 404 returned error can't find the container with id e40b439837fe1d91457b99d2a10d89fc3576a663d611a4130e1cd75fbc627db8 Sep 05 06:40:58.060377 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:58.060355 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw"] Sep 05 06:40:58.063087 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:40:58.063061 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2991e799_10c4_4bd8_9a2a_ec2a94350816.slice/crio-7c5f4615e60c2943597109429b31f00936fd6111b91e8e54dd1d4a3fa2836e88 WatchSource:0}: Error finding container 7c5f4615e60c2943597109429b31f00936fd6111b91e8e54dd1d4a3fa2836e88: Status 404 returned error can't find the container with id 7c5f4615e60c2943597109429b31f00936fd6111b91e8e54dd1d4a3fa2836e88 Sep 05 06:40:58.509863 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:58.509821 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw" event={"ID":"2991e799-10c4-4bd8-9a2a-ec2a94350816","Type":"ContainerStarted","Data":"893bc50f08a8964f23fd7e6d5946e62da6fa173c126fb31e7a869e0e10ee8169"} Sep 05 06:40:58.510359 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:58.509892 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw" event={"ID":"2991e799-10c4-4bd8-9a2a-ec2a94350816","Type":"ContainerStarted","Data":"7c5f4615e60c2943597109429b31f00936fd6111b91e8e54dd1d4a3fa2836e88"} Sep 05 06:40:58.510359 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:58.510012 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw" Sep 05 06:40:58.511384 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:58.511363 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd" event={"ID":"52449020-e29a-4edd-9f20-d0a2ca28d380","Type":"ContainerStarted","Data":"9d52389987acd22e28e83096dc4454174db5c1d7b4f8ceda6ab24cab154e37be"} Sep 05 06:40:58.511510 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:58.511402 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd" event={"ID":"52449020-e29a-4edd-9f20-d0a2ca28d380","Type":"ContainerStarted","Data":"e40b439837fe1d91457b99d2a10d89fc3576a663d611a4130e1cd75fbc627db8"} Sep 05 06:40:58.511510 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:58.511495 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd" Sep 05 06:40:58.536113 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:58.536037 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw" podStartSLOduration=1.536016302 podStartE2EDuration="1.536016302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:40:58.532443031 +0000 UTC m=+882.916986155" watchObservedRunningTime="2026-09-05 06:40:58.536016302 +0000 UTC m=+882.920559429" Sep 05 06:40:58.554030 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:40:58.553979 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd" podStartSLOduration=1.553961334 podStartE2EDuration="1.553961334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:40:58.55327002 +0000 UTC m=+882.937813143" watchObservedRunningTime="2026-09-05 06:40:58.553961334 +0000 UTC m=+882.938504457" Sep 05 06:41:09.518518 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:09.518444 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd" Sep 05 06:41:09.518518 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:09.518514 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw" Sep 05 06:41:19.913568 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:19.913522 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13503c3c-5cj52"] Sep 05 06:41:19.918584 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:19.918559 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13503c3c-5cj52" Sep 05 06:41:19.920849 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:19.920824 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slfqj\"" Sep 05 06:41:19.920998 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:19.920824 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 05 06:41:19.924737 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:19.924711 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13503c3c-5cj52"] Sep 05 06:41:20.012407 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:20.012367 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e5694f6-6709-4e78-b29d-faba19015790-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-13503c3c-5cj52\" (UID: \"2e5694f6-6709-4e78-b29d-faba19015790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13503c3c-5cj52" Sep 05 06:41:20.113791 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:20.113753 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e5694f6-6709-4e78-b29d-faba19015790-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-13503c3c-5cj52\" (UID: \"2e5694f6-6709-4e78-b29d-faba19015790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13503c3c-5cj52" Sep 05 06:41:20.114381 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:20.114362 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e5694f6-6709-4e78-b29d-faba19015790-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-13503c3c-5cj52\" (UID: \"2e5694f6-6709-4e78-b29d-faba19015790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13503c3c-5cj52" Sep 05 06:41:20.232020 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:20.231926 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13503c3c-5cj52" Sep 05 06:41:20.506133 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:20.506105 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13503c3c-5cj52"] Sep 05 06:41:20.508472 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:41:20.508439 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5694f6_6709_4e78_b29d_faba19015790.slice/crio-5fbfd35495a2e5c1a056f20a3c07de5a3e8b4b2c41bb2ec5fd55b2f8b5f4ebbc WatchSource:0}: Error finding container 5fbfd35495a2e5c1a056f20a3c07de5a3e8b4b2c41bb2ec5fd55b2f8b5f4ebbc: Status 404 returned error can't find the container with id 5fbfd35495a2e5c1a056f20a3c07de5a3e8b4b2c41bb2ec5fd55b2f8b5f4ebbc Sep 05 06:41:20.651811 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:20.651774 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13503c3c-5cj52" event={"ID":"2e5694f6-6709-4e78-b29d-faba19015790","Type":"ContainerStarted","Data":"5fbfd35495a2e5c1a056f20a3c07de5a3e8b4b2c41bb2ec5fd55b2f8b5f4ebbc"} Sep 05 06:41:21.658655 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:21.658623 2565 generic.go:356] "Generic (PLEG): container finished" podID="2e5694f6-6709-4e78-b29d-faba19015790" containerID="3b45e7fbc25ed6fa33083958bfc33bf5d680f253e55238289a889b84a2e14777" exitCode=0 Sep 05 06:41:21.659044 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:21.658714 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13503c3c-5cj52" event={"ID":"2e5694f6-6709-4e78-b29d-faba19015790","Type":"ContainerDied","Data":"3b45e7fbc25ed6fa33083958bfc33bf5d680f253e55238289a889b84a2e14777"} Sep 05 06:41:22.802192 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:22.802166 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13503c3c-5cj52" Sep 05 06:41:22.943722 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:22.943637 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e5694f6-6709-4e78-b29d-faba19015790-maas-api-service-ca\") pod \"2e5694f6-6709-4e78-b29d-faba19015790\" (UID: \"2e5694f6-6709-4e78-b29d-faba19015790\") " Sep 05 06:41:22.943988 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:22.943963 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e5694f6-6709-4e78-b29d-faba19015790-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2e5694f6-6709-4e78-b29d-faba19015790" (UID: "2e5694f6-6709-4e78-b29d-faba19015790"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:41:23.045209 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:23.045170 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e5694f6-6709-4e78-b29d-faba19015790-maas-api-service-ca\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:41:23.670299 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:23.670267 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13503c3c-5cj52" Sep 05 06:41:23.670299 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:23.670269 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13503c3c-5cj52" event={"ID":"2e5694f6-6709-4e78-b29d-faba19015790","Type":"ContainerDied","Data":"5fbfd35495a2e5c1a056f20a3c07de5a3e8b4b2c41bb2ec5fd55b2f8b5f4ebbc"} Sep 05 06:41:23.670538 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:23.670369 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fbfd35495a2e5c1a056f20a3c07de5a3e8b4b2c41bb2ec5fd55b2f8b5f4ebbc" Sep 05 06:41:27.011955 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.011911 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw"] Sep 05 06:41:27.012470 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.012198 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw" podUID="2991e799-10c4-4bd8-9a2a-ec2a94350816" containerName="payload-processing" containerID="cri-o://893bc50f08a8964f23fd7e6d5946e62da6fa173c126fb31e7a869e0e10ee8169" gracePeriod=30 Sep 05 06:41:27.018575 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.018545 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd"] Sep 05 06:41:27.018868 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.018813 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd" podUID="52449020-e29a-4edd-9f20-d0a2ca28d380" containerName="payload-pre-processing" containerID="cri-o://9d52389987acd22e28e83096dc4454174db5c1d7b4f8ceda6ab24cab154e37be" gracePeriod=30 Sep 05 06:41:27.223364 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.223331 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13503c3c-5cj52"] Sep 05 06:41:27.240473 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.240439 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13503c3c-5cj52"] Sep 05 06:41:27.298695 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.298670 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw" Sep 05 06:41:27.301635 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.301612 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd" Sep 05 06:41:27.393617 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.393586 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47jwl\" (UniqueName: \"kubernetes.io/projected/2991e799-10c4-4bd8-9a2a-ec2a94350816-kube-api-access-47jwl\") pod \"2991e799-10c4-4bd8-9a2a-ec2a94350816\" (UID: \"2991e799-10c4-4bd8-9a2a-ec2a94350816\") " Sep 05 06:41:27.393806 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.393652 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52449020-e29a-4edd-9f20-d0a2ca28d380-plugins-config-volume\") pod \"52449020-e29a-4edd-9f20-d0a2ca28d380\" (UID: \"52449020-e29a-4edd-9f20-d0a2ca28d380\") " Sep 05 06:41:27.393806 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.393718 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwndf\" (UniqueName: \"kubernetes.io/projected/52449020-e29a-4edd-9f20-d0a2ca28d380-kube-api-access-dwndf\") pod \"52449020-e29a-4edd-9f20-d0a2ca28d380\" (UID: \"52449020-e29a-4edd-9f20-d0a2ca28d380\") " Sep 05 06:41:27.393937 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.393806 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2991e799-10c4-4bd8-9a2a-ec2a94350816-plugins-config-volume\") pod \"2991e799-10c4-4bd8-9a2a-ec2a94350816\" (UID: \"2991e799-10c4-4bd8-9a2a-ec2a94350816\") " Sep 05 06:41:27.394226 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.394047 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52449020-e29a-4edd-9f20-d0a2ca28d380-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "52449020-e29a-4edd-9f20-d0a2ca28d380" (UID: "52449020-e29a-4edd-9f20-d0a2ca28d380"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:41:27.394446 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.394397 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2991e799-10c4-4bd8-9a2a-ec2a94350816-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2991e799-10c4-4bd8-9a2a-ec2a94350816" (UID: "2991e799-10c4-4bd8-9a2a-ec2a94350816"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:41:27.394844 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.394823 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52449020-e29a-4edd-9f20-d0a2ca28d380-plugins-config-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:41:27.394936 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.394868 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2991e799-10c4-4bd8-9a2a-ec2a94350816-plugins-config-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:41:27.396285 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.396263 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2991e799-10c4-4bd8-9a2a-ec2a94350816-kube-api-access-47jwl" (OuterVolumeSpecName: "kube-api-access-47jwl") pod "2991e799-10c4-4bd8-9a2a-ec2a94350816" (UID: "2991e799-10c4-4bd8-9a2a-ec2a94350816"). InnerVolumeSpecName "kube-api-access-47jwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:41:27.396417 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.396398 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52449020-e29a-4edd-9f20-d0a2ca28d380-kube-api-access-dwndf" (OuterVolumeSpecName: "kube-api-access-dwndf") pod "52449020-e29a-4edd-9f20-d0a2ca28d380" (UID: "52449020-e29a-4edd-9f20-d0a2ca28d380"). InnerVolumeSpecName "kube-api-access-dwndf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:41:27.495498 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.495460 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwndf\" (UniqueName: \"kubernetes.io/projected/52449020-e29a-4edd-9f20-d0a2ca28d380-kube-api-access-dwndf\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:41:27.495498 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.495493 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47jwl\" (UniqueName: \"kubernetes.io/projected/2991e799-10c4-4bd8-9a2a-ec2a94350816-kube-api-access-47jwl\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:41:27.690977 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.690939 2565 generic.go:356] "Generic (PLEG): container finished" podID="2991e799-10c4-4bd8-9a2a-ec2a94350816" containerID="893bc50f08a8964f23fd7e6d5946e62da6fa173c126fb31e7a869e0e10ee8169" exitCode=0 Sep 05 06:41:27.691158 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.690989 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw" Sep 05 06:41:27.691158 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.691018 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw" event={"ID":"2991e799-10c4-4bd8-9a2a-ec2a94350816","Type":"ContainerDied","Data":"893bc50f08a8964f23fd7e6d5946e62da6fa173c126fb31e7a869e0e10ee8169"} Sep 05 06:41:27.691158 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.691074 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw" event={"ID":"2991e799-10c4-4bd8-9a2a-ec2a94350816","Type":"ContainerDied","Data":"7c5f4615e60c2943597109429b31f00936fd6111b91e8e54dd1d4a3fa2836e88"} Sep 05 06:41:27.691158 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.691100 2565 scope.go:117] "RemoveContainer" containerID="893bc50f08a8964f23fd7e6d5946e62da6fa173c126fb31e7a869e0e10ee8169" Sep 05 06:41:27.692500 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.692476 2565 generic.go:356] "Generic (PLEG): container finished" podID="52449020-e29a-4edd-9f20-d0a2ca28d380" containerID="9d52389987acd22e28e83096dc4454174db5c1d7b4f8ceda6ab24cab154e37be" exitCode=0 Sep 05 06:41:27.692636 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.692508 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd" Sep 05 06:41:27.692636 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.692511 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd" event={"ID":"52449020-e29a-4edd-9f20-d0a2ca28d380","Type":"ContainerDied","Data":"9d52389987acd22e28e83096dc4454174db5c1d7b4f8ceda6ab24cab154e37be"} Sep 05 06:41:27.692636 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.692614 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd" event={"ID":"52449020-e29a-4edd-9f20-d0a2ca28d380","Type":"ContainerDied","Data":"e40b439837fe1d91457b99d2a10d89fc3576a663d611a4130e1cd75fbc627db8"} Sep 05 06:41:27.701217 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.701198 2565 scope.go:117] "RemoveContainer" containerID="893bc50f08a8964f23fd7e6d5946e62da6fa173c126fb31e7a869e0e10ee8169" Sep 05 06:41:27.701594 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:41:27.701576 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"893bc50f08a8964f23fd7e6d5946e62da6fa173c126fb31e7a869e0e10ee8169\": container with ID starting with 893bc50f08a8964f23fd7e6d5946e62da6fa173c126fb31e7a869e0e10ee8169 not found: ID does not exist" containerID="893bc50f08a8964f23fd7e6d5946e62da6fa173c126fb31e7a869e0e10ee8169" Sep 05 06:41:27.701648 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.701603 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"893bc50f08a8964f23fd7e6d5946e62da6fa173c126fb31e7a869e0e10ee8169"} err="failed to get container status \"893bc50f08a8964f23fd7e6d5946e62da6fa173c126fb31e7a869e0e10ee8169\": rpc error: code = NotFound desc = could not find container \"893bc50f08a8964f23fd7e6d5946e62da6fa173c126fb31e7a869e0e10ee8169\": container with ID starting with 893bc50f08a8964f23fd7e6d5946e62da6fa173c126fb31e7a869e0e10ee8169 not found: ID does not exist" Sep 05 06:41:27.701648 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.701619 2565 scope.go:117] "RemoveContainer" containerID="9d52389987acd22e28e83096dc4454174db5c1d7b4f8ceda6ab24cab154e37be" Sep 05 06:41:27.709858 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.709837 2565 scope.go:117] "RemoveContainer" containerID="9d52389987acd22e28e83096dc4454174db5c1d7b4f8ceda6ab24cab154e37be" Sep 05 06:41:27.710172 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:41:27.710154 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d52389987acd22e28e83096dc4454174db5c1d7b4f8ceda6ab24cab154e37be\": container with ID starting with 9d52389987acd22e28e83096dc4454174db5c1d7b4f8ceda6ab24cab154e37be not found: ID does not exist" containerID="9d52389987acd22e28e83096dc4454174db5c1d7b4f8ceda6ab24cab154e37be" Sep 05 06:41:27.710234 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.710179 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d52389987acd22e28e83096dc4454174db5c1d7b4f8ceda6ab24cab154e37be"} err="failed to get container status \"9d52389987acd22e28e83096dc4454174db5c1d7b4f8ceda6ab24cab154e37be\": rpc error: code = NotFound desc = could not find container \"9d52389987acd22e28e83096dc4454174db5c1d7b4f8ceda6ab24cab154e37be\": container with ID starting with 9d52389987acd22e28e83096dc4454174db5c1d7b4f8ceda6ab24cab154e37be not found: ID does not exist" Sep 05 06:41:27.719001 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.718977 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw"] Sep 05 06:41:27.721187 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.721164 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-13503c3c-6f5b897c5c-m8cmw"] Sep 05 06:41:27.733186 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.733162 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd"] Sep 05 06:41:27.741985 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:27.741961 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-13503c3c-66f4bb6494-wdzvd"] Sep 05 06:41:28.165961 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:28.165925 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2991e799-10c4-4bd8-9a2a-ec2a94350816" path="/var/lib/kubelet/pods/2991e799-10c4-4bd8-9a2a-ec2a94350816/volumes" Sep 05 06:41:28.172466 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:28.166722 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e5694f6-6709-4e78-b29d-faba19015790" path="/var/lib/kubelet/pods/2e5694f6-6709-4e78-b29d-faba19015790/volumes" Sep 05 06:41:28.172466 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:28.167335 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52449020-e29a-4edd-9f20-d0a2ca28d380" path="/var/lib/kubelet/pods/52449020-e29a-4edd-9f20-d0a2ca28d380/volumes" Sep 05 06:41:57.403195 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:57.403150 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9f698c-wmpfh"] Sep 05 06:41:57.403792 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:57.403771 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2991e799-10c4-4bd8-9a2a-ec2a94350816" containerName="payload-processing" Sep 05 06:41:57.403881 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:57.403795 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2991e799-10c4-4bd8-9a2a-ec2a94350816" containerName="payload-processing" Sep 05 06:41:57.403881 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:57.403827 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e5694f6-6709-4e78-b29d-faba19015790" containerName="revoke-keys" Sep 05 06:41:57.403881 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:57.403838 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e5694f6-6709-4e78-b29d-faba19015790" containerName="revoke-keys" Sep 05 06:41:57.403881 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:57.403869 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52449020-e29a-4edd-9f20-d0a2ca28d380" containerName="payload-pre-processing" Sep 05 06:41:57.403881 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:57.403878 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="52449020-e29a-4edd-9f20-d0a2ca28d380" containerName="payload-pre-processing" Sep 05 06:41:57.404111 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:57.403977 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e5694f6-6709-4e78-b29d-faba19015790" containerName="revoke-keys" Sep 05 06:41:57.404111 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:57.403991 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="52449020-e29a-4edd-9f20-d0a2ca28d380" containerName="payload-pre-processing" Sep 05 06:41:57.404111 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:57.404004 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2991e799-10c4-4bd8-9a2a-ec2a94350816" containerName="payload-processing" Sep 05 06:41:57.407249 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:57.407228 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9f698c-wmpfh" Sep 05 06:41:57.409660 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:57.409640 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 05 06:41:57.409660 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:57.409653 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slfqj\"" Sep 05 06:41:57.415377 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:57.415353 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9f698c-wmpfh"] Sep 05 06:41:57.480005 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:57.479962 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ee6846c-5eb3-4653-ba09-a31e8b3f9e89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-9f698c-wmpfh\" (UID: \"6ee6846c-5eb3-4653-ba09-a31e8b3f9e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9f698c-wmpfh" Sep 05 06:41:57.581233 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:57.581193 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ee6846c-5eb3-4653-ba09-a31e8b3f9e89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-9f698c-wmpfh\" (UID: \"6ee6846c-5eb3-4653-ba09-a31e8b3f9e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9f698c-wmpfh" Sep 05 06:41:57.581903 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:57.581877 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ee6846c-5eb3-4653-ba09-a31e8b3f9e89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-9f698c-wmpfh\" (UID: \"6ee6846c-5eb3-4653-ba09-a31e8b3f9e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9f698c-wmpfh" Sep 05 06:41:57.720519 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:57.720428 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9f698c-wmpfh" Sep 05 06:41:57.993793 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:57.993699 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9f698c-wmpfh"] Sep 05 06:41:57.996630 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:41:57.996591 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee6846c_5eb3_4653_ba09_a31e8b3f9e89.slice/crio-a26bc35d1f44c20ec0e86e966cbb63996df811dee2c483429d7a99d0107cd00f WatchSource:0}: Error finding container a26bc35d1f44c20ec0e86e966cbb63996df811dee2c483429d7a99d0107cd00f: Status 404 returned error can't find the container with id a26bc35d1f44c20ec0e86e966cbb63996df811dee2c483429d7a99d0107cd00f Sep 05 06:41:58.856215 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:58.856183 2565 generic.go:356] "Generic (PLEG): container finished" podID="6ee6846c-5eb3-4653-ba09-a31e8b3f9e89" containerID="6eec886018eaa4662575e8f0a04c62e3a797890462e2e62d5281ff1b239c0d9a" exitCode=0 Sep 05 06:41:58.856623 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:58.856274 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9f698c-wmpfh" event={"ID":"6ee6846c-5eb3-4653-ba09-a31e8b3f9e89","Type":"ContainerDied","Data":"6eec886018eaa4662575e8f0a04c62e3a797890462e2e62d5281ff1b239c0d9a"} Sep 05 06:41:58.856623 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:58.856344 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9f698c-wmpfh" event={"ID":"6ee6846c-5eb3-4653-ba09-a31e8b3f9e89","Type":"ContainerStarted","Data":"a26bc35d1f44c20ec0e86e966cbb63996df811dee2c483429d7a99d0107cd00f"} Sep 05 06:41:59.997898 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:41:59.997870 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9f698c-wmpfh" Sep 05 06:42:00.106414 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:00.106365 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ee6846c-5eb3-4653-ba09-a31e8b3f9e89-maas-api-service-ca\") pod \"6ee6846c-5eb3-4653-ba09-a31e8b3f9e89\" (UID: \"6ee6846c-5eb3-4653-ba09-a31e8b3f9e89\") " Sep 05 06:42:00.106720 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:00.106698 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ee6846c-5eb3-4653-ba09-a31e8b3f9e89-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6ee6846c-5eb3-4653-ba09-a31e8b3f9e89" (UID: "6ee6846c-5eb3-4653-ba09-a31e8b3f9e89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:42:00.207985 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:00.207941 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ee6846c-5eb3-4653-ba09-a31e8b3f9e89-maas-api-service-ca\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:42:00.867683 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:00.867650 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9f698c-wmpfh" Sep 05 06:42:00.867861 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:00.867682 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9f698c-wmpfh" event={"ID":"6ee6846c-5eb3-4653-ba09-a31e8b3f9e89","Type":"ContainerDied","Data":"a26bc35d1f44c20ec0e86e966cbb63996df811dee2c483429d7a99d0107cd00f"} Sep 05 06:42:00.867861 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:00.867731 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a26bc35d1f44c20ec0e86e966cbb63996df811dee2c483429d7a99d0107cd00f" Sep 05 06:42:08.157857 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:08.157823 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9f698c-wmpfh"] Sep 05 06:42:08.165968 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:08.165939 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9f698c-wmpfh"] Sep 05 06:42:10.165825 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:10.165790 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ee6846c-5eb3-4653-ba09-a31e8b3f9e89" path="/var/lib/kubelet/pods/6ee6846c-5eb3-4653-ba09-a31e8b3f9e89/volumes" Sep 05 06:42:55.203944 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:55.203907 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e697-52xj7"] Sep 05 06:42:55.204612 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:55.204592 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ee6846c-5eb3-4653-ba09-a31e8b3f9e89" containerName="revoke-keys" Sep 05 06:42:55.204700 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:55.204615 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ee6846c-5eb3-4653-ba09-a31e8b3f9e89" containerName="revoke-keys" Sep 05 06:42:55.204753 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:55.204710 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ee6846c-5eb3-4653-ba09-a31e8b3f9e89" containerName="revoke-keys" Sep 05 06:42:55.207925 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:55.207904 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e697-52xj7" Sep 05 06:42:55.210202 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:55.210181 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slfqj\"" Sep 05 06:42:55.210326 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:55.210181 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 05 06:42:55.216539 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:55.216518 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e697-52xj7"] Sep 05 06:42:55.263877 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:55.263838 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11110dbf-1b9c-4d0c-b014-ba2822faf830-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-45e697-52xj7\" (UID: \"11110dbf-1b9c-4d0c-b014-ba2822faf830\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e697-52xj7" Sep 05 06:42:55.364537 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:55.364500 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11110dbf-1b9c-4d0c-b014-ba2822faf830-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-45e697-52xj7\" (UID: \"11110dbf-1b9c-4d0c-b014-ba2822faf830\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e697-52xj7" Sep 05 06:42:55.365132 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:55.365112 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11110dbf-1b9c-4d0c-b014-ba2822faf830-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-45e697-52xj7\" (UID: \"11110dbf-1b9c-4d0c-b014-ba2822faf830\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e697-52xj7" Sep 05 06:42:55.520038 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:55.519940 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e697-52xj7" Sep 05 06:42:55.792977 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:55.792953 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e697-52xj7"] Sep 05 06:42:55.795740 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:42:55.795701 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11110dbf_1b9c_4d0c_b014_ba2822faf830.slice/crio-2c694c7897cbbf6f801ea64c2cd78c95a7cf01d1e7a8b1d4953e2f61ba80de1d WatchSource:0}: Error finding container 2c694c7897cbbf6f801ea64c2cd78c95a7cf01d1e7a8b1d4953e2f61ba80de1d: Status 404 returned error can't find the container with id 2c694c7897cbbf6f801ea64c2cd78c95a7cf01d1e7a8b1d4953e2f61ba80de1d Sep 05 06:42:56.183571 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:56.183523 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e697-52xj7" event={"ID":"11110dbf-1b9c-4d0c-b014-ba2822faf830","Type":"ContainerStarted","Data":"afdc8457567f6cd2bf897248b6b40c97593db75bc2c138a713839e06ead0b916"} Sep 05 06:42:56.183745 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:56.183586 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e697-52xj7" event={"ID":"11110dbf-1b9c-4d0c-b014-ba2822faf830","Type":"ContainerStarted","Data":"2c694c7897cbbf6f801ea64c2cd78c95a7cf01d1e7a8b1d4953e2f61ba80de1d"} Sep 05 06:42:56.200641 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:56.200588 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e697-52xj7" podStartSLOduration=1.2005718220000001 podStartE2EDuration="1.200571822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:42:56.199269734 +0000 UTC m=+1000.583812857" watchObservedRunningTime="2026-09-05 06:42:56.200571822 +0000 UTC m=+1000.585114946" Sep 05 06:42:57.189782 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:57.189749 2565 generic.go:356] "Generic (PLEG): container finished" podID="11110dbf-1b9c-4d0c-b014-ba2822faf830" containerID="afdc8457567f6cd2bf897248b6b40c97593db75bc2c138a713839e06ead0b916" exitCode=0 Sep 05 06:42:57.190171 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:57.189839 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e697-52xj7" event={"ID":"11110dbf-1b9c-4d0c-b014-ba2822faf830","Type":"ContainerDied","Data":"afdc8457567f6cd2bf897248b6b40c97593db75bc2c138a713839e06ead0b916"} Sep 05 06:42:58.331868 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:58.331835 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e697-52xj7" Sep 05 06:42:58.396550 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:58.396522 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11110dbf-1b9c-4d0c-b014-ba2822faf830-maas-api-service-ca\") pod \"11110dbf-1b9c-4d0c-b014-ba2822faf830\" (UID: \"11110dbf-1b9c-4d0c-b014-ba2822faf830\") " Sep 05 06:42:58.396862 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:58.396836 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11110dbf-1b9c-4d0c-b014-ba2822faf830-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "11110dbf-1b9c-4d0c-b014-ba2822faf830" (UID: "11110dbf-1b9c-4d0c-b014-ba2822faf830"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:42:58.497546 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:58.497464 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11110dbf-1b9c-4d0c-b014-ba2822faf830-maas-api-service-ca\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:42:59.201631 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:59.201598 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e697-52xj7" Sep 05 06:42:59.201817 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:59.201629 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e697-52xj7" event={"ID":"11110dbf-1b9c-4d0c-b014-ba2822faf830","Type":"ContainerDied","Data":"2c694c7897cbbf6f801ea64c2cd78c95a7cf01d1e7a8b1d4953e2f61ba80de1d"} Sep 05 06:42:59.201817 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:42:59.201706 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c694c7897cbbf6f801ea64c2cd78c95a7cf01d1e7a8b1d4953e2f61ba80de1d" Sep 05 06:43:05.871158 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:05.871125 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p"] Sep 05 06:43:05.871743 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:05.871419 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p" podUID="01f57470-5045-42eb-91c1-1734be3eeadb" containerName="payload-processing" containerID="cri-o://962340836d5c849da3a52f2ab2ee07b88b00d2285453626a096dd6610951391a" gracePeriod=30 Sep 05 06:43:05.877764 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:05.877734 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg"] Sep 05 06:43:05.878045 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:05.878017 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg" podUID="ef869309-90f5-472c-ad55-035a0d36f774" containerName="payload-pre-processing" containerID="cri-o://283d2b8b7de9597801ac739d9452fe2b6975ffacfe4eae056420793e56f6ff3e" gracePeriod=30 Sep 05 06:43:06.041807 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.041775 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e697-52xj7"] Sep 05 06:43:06.047483 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.047396 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e697-52xj7"] Sep 05 06:43:06.152147 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.152121 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p" Sep 05 06:43:06.154806 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.154784 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg" Sep 05 06:43:06.171424 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.171390 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11110dbf-1b9c-4d0c-b014-ba2822faf830" path="/var/lib/kubelet/pods/11110dbf-1b9c-4d0c-b014-ba2822faf830/volumes" Sep 05 06:43:06.242070 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.242031 2565 generic.go:356] "Generic (PLEG): container finished" podID="01f57470-5045-42eb-91c1-1734be3eeadb" containerID="962340836d5c849da3a52f2ab2ee07b88b00d2285453626a096dd6610951391a" exitCode=0 Sep 05 06:43:06.242269 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.242095 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p" Sep 05 06:43:06.242269 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.242113 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p" event={"ID":"01f57470-5045-42eb-91c1-1734be3eeadb","Type":"ContainerDied","Data":"962340836d5c849da3a52f2ab2ee07b88b00d2285453626a096dd6610951391a"} Sep 05 06:43:06.242269 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.242176 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p" event={"ID":"01f57470-5045-42eb-91c1-1734be3eeadb","Type":"ContainerDied","Data":"87ce7af81c27016ee352f4061cd6abe43a952689753f7f1003d4b382bc6abdc8"} Sep 05 06:43:06.242269 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.242206 2565 scope.go:117] "RemoveContainer" containerID="962340836d5c849da3a52f2ab2ee07b88b00d2285453626a096dd6610951391a" Sep 05 06:43:06.243583 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.243561 2565 generic.go:356] "Generic (PLEG): container finished" podID="ef869309-90f5-472c-ad55-035a0d36f774" containerID="283d2b8b7de9597801ac739d9452fe2b6975ffacfe4eae056420793e56f6ff3e" exitCode=0 Sep 05 06:43:06.243689 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.243617 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg" event={"ID":"ef869309-90f5-472c-ad55-035a0d36f774","Type":"ContainerDied","Data":"283d2b8b7de9597801ac739d9452fe2b6975ffacfe4eae056420793e56f6ff3e"} Sep 05 06:43:06.243689 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.243625 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg" Sep 05 06:43:06.243689 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.243657 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg" event={"ID":"ef869309-90f5-472c-ad55-035a0d36f774","Type":"ContainerDied","Data":"1ef8a494a2855a132f82c584286f374fbf32a3a6510e5b094f20d51f51e145f2"} Sep 05 06:43:06.251694 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.251675 2565 scope.go:117] "RemoveContainer" containerID="962340836d5c849da3a52f2ab2ee07b88b00d2285453626a096dd6610951391a" Sep 05 06:43:06.251985 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:43:06.251967 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"962340836d5c849da3a52f2ab2ee07b88b00d2285453626a096dd6610951391a\": container with ID starting with 962340836d5c849da3a52f2ab2ee07b88b00d2285453626a096dd6610951391a not found: ID does not exist" containerID="962340836d5c849da3a52f2ab2ee07b88b00d2285453626a096dd6610951391a" Sep 05 06:43:06.252041 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.251993 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962340836d5c849da3a52f2ab2ee07b88b00d2285453626a096dd6610951391a"} err="failed to get container status \"962340836d5c849da3a52f2ab2ee07b88b00d2285453626a096dd6610951391a\": rpc error: code = NotFound desc = could not find container \"962340836d5c849da3a52f2ab2ee07b88b00d2285453626a096dd6610951391a\": container with ID starting with 962340836d5c849da3a52f2ab2ee07b88b00d2285453626a096dd6610951391a not found: ID does not exist" Sep 05 06:43:06.252041 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.252012 2565 scope.go:117] "RemoveContainer" containerID="283d2b8b7de9597801ac739d9452fe2b6975ffacfe4eae056420793e56f6ff3e" Sep 05 06:43:06.259965 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.259951 2565 scope.go:117] "RemoveContainer" containerID="283d2b8b7de9597801ac739d9452fe2b6975ffacfe4eae056420793e56f6ff3e" Sep 05 06:43:06.260222 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:43:06.260204 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"283d2b8b7de9597801ac739d9452fe2b6975ffacfe4eae056420793e56f6ff3e\": container with ID starting with 283d2b8b7de9597801ac739d9452fe2b6975ffacfe4eae056420793e56f6ff3e not found: ID does not exist" containerID="283d2b8b7de9597801ac739d9452fe2b6975ffacfe4eae056420793e56f6ff3e" Sep 05 06:43:06.260279 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.260233 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"283d2b8b7de9597801ac739d9452fe2b6975ffacfe4eae056420793e56f6ff3e"} err="failed to get container status \"283d2b8b7de9597801ac739d9452fe2b6975ffacfe4eae056420793e56f6ff3e\": rpc error: code = NotFound desc = could not find container \"283d2b8b7de9597801ac739d9452fe2b6975ffacfe4eae056420793e56f6ff3e\": container with ID starting with 283d2b8b7de9597801ac739d9452fe2b6975ffacfe4eae056420793e56f6ff3e not found: ID does not exist" Sep 05 06:43:06.276583 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.276556 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpjn7\" (UniqueName: \"kubernetes.io/projected/01f57470-5045-42eb-91c1-1734be3eeadb-kube-api-access-fpjn7\") pod \"01f57470-5045-42eb-91c1-1734be3eeadb\" (UID: \"01f57470-5045-42eb-91c1-1734be3eeadb\") " Sep 05 06:43:06.276670 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.276616 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwrds\" (UniqueName: \"kubernetes.io/projected/ef869309-90f5-472c-ad55-035a0d36f774-kube-api-access-nwrds\") pod \"ef869309-90f5-472c-ad55-035a0d36f774\" (UID: \"ef869309-90f5-472c-ad55-035a0d36f774\") " Sep 05 06:43:06.276670 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.276657 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f57470-5045-42eb-91c1-1734be3eeadb-plugins-config-volume\") pod \"01f57470-5045-42eb-91c1-1734be3eeadb\" (UID: \"01f57470-5045-42eb-91c1-1734be3eeadb\") " Sep 05 06:43:06.276762 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.276699 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef869309-90f5-472c-ad55-035a0d36f774-plugins-config-volume\") pod \"ef869309-90f5-472c-ad55-035a0d36f774\" (UID: \"ef869309-90f5-472c-ad55-035a0d36f774\") " Sep 05 06:43:06.277024 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.277002 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f57470-5045-42eb-91c1-1734be3eeadb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "01f57470-5045-42eb-91c1-1734be3eeadb" (UID: "01f57470-5045-42eb-91c1-1734be3eeadb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:43:06.277096 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.277077 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef869309-90f5-472c-ad55-035a0d36f774-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ef869309-90f5-472c-ad55-035a0d36f774" (UID: "ef869309-90f5-472c-ad55-035a0d36f774"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:43:06.278817 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.278796 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef869309-90f5-472c-ad55-035a0d36f774-kube-api-access-nwrds" (OuterVolumeSpecName: "kube-api-access-nwrds") pod "ef869309-90f5-472c-ad55-035a0d36f774" (UID: "ef869309-90f5-472c-ad55-035a0d36f774"). InnerVolumeSpecName "kube-api-access-nwrds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:43:06.278873 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.278821 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f57470-5045-42eb-91c1-1734be3eeadb-kube-api-access-fpjn7" (OuterVolumeSpecName: "kube-api-access-fpjn7") pod "01f57470-5045-42eb-91c1-1734be3eeadb" (UID: "01f57470-5045-42eb-91c1-1734be3eeadb"). InnerVolumeSpecName "kube-api-access-fpjn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:43:06.378515 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.378426 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpjn7\" (UniqueName: \"kubernetes.io/projected/01f57470-5045-42eb-91c1-1734be3eeadb-kube-api-access-fpjn7\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:06.378515 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.378455 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwrds\" (UniqueName: \"kubernetes.io/projected/ef869309-90f5-472c-ad55-035a0d36f774-kube-api-access-nwrds\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:06.378515 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.378464 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f57470-5045-42eb-91c1-1734be3eeadb-plugins-config-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:06.378515 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.378473 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef869309-90f5-472c-ad55-035a0d36f774-plugins-config-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:06.568026 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.567995 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p"] Sep 05 06:43:06.571764 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.571738 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-45e697-fdb4f9557-lpb6p"] Sep 05 06:43:06.584024 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.583997 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg"] Sep 05 06:43:06.589011 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:06.588989 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-45e697-c4b56b586-6csqg"] Sep 05 06:43:08.165693 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:08.165660 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01f57470-5045-42eb-91c1-1734be3eeadb" path="/var/lib/kubelet/pods/01f57470-5045-42eb-91c1-1734be3eeadb/volumes" Sep 05 06:43:08.166051 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:08.166025 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef869309-90f5-472c-ad55-035a0d36f774" path="/var/lib/kubelet/pods/ef869309-90f5-472c-ad55-035a0d36f774/volumes" Sep 05 06:43:11.442201 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.442166 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw"] Sep 05 06:43:11.442676 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.442594 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01f57470-5045-42eb-91c1-1734be3eeadb" containerName="payload-processing" Sep 05 06:43:11.442676 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.442607 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="01f57470-5045-42eb-91c1-1734be3eeadb" containerName="payload-processing" Sep 05 06:43:11.442676 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.442625 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11110dbf-1b9c-4d0c-b014-ba2822faf830" containerName="revoke-keys" Sep 05 06:43:11.442676 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.442631 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="11110dbf-1b9c-4d0c-b014-ba2822faf830" containerName="revoke-keys" Sep 05 06:43:11.442676 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.442642 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef869309-90f5-472c-ad55-035a0d36f774" containerName="payload-pre-processing" Sep 05 06:43:11.442676 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.442648 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef869309-90f5-472c-ad55-035a0d36f774" containerName="payload-pre-processing" Sep 05 06:43:11.442910 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.442702 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="11110dbf-1b9c-4d0c-b014-ba2822faf830" containerName="revoke-keys" Sep 05 06:43:11.442910 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.442712 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="01f57470-5045-42eb-91c1-1734be3eeadb" containerName="payload-processing" Sep 05 06:43:11.442910 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.442719 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef869309-90f5-472c-ad55-035a0d36f774" containerName="payload-pre-processing" Sep 05 06:43:11.445773 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.445758 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.448607 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.448587 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-810a2bb6-openshift-default-dockercfg-92htx\"" Sep 05 06:43:11.458062 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.458035 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw"] Sep 05 06:43:11.529806 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.528785 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e28eb1-8011-4e71-a427-07c65bf41750-istiod-ca-cert\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.529806 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.528888 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-istio-data\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.529806 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.528946 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-credential-socket\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.529806 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.528978 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e28eb1-8011-4e71-a427-07c65bf41750-istio-podinfo\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.529806 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.529015 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-workload-socket\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.529806 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.529074 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e28eb1-8011-4e71-a427-07c65bf41750-istio-token\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.529806 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.529105 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-workload-certs\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.529806 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.529138 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhn9v\" (UniqueName: \"kubernetes.io/projected/99e28eb1-8011-4e71-a427-07c65bf41750-kube-api-access-zhn9v\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.529806 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.529174 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-istio-envoy\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.630326 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.630273 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-workload-certs\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.630497 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.630338 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhn9v\" (UniqueName: \"kubernetes.io/projected/99e28eb1-8011-4e71-a427-07c65bf41750-kube-api-access-zhn9v\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.630497 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.630372 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-istio-envoy\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.630497 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.630451 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e28eb1-8011-4e71-a427-07c65bf41750-istiod-ca-cert\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.630497 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.630490 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-istio-data\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.630665 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.630532 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-credential-socket\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.630665 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.630564 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e28eb1-8011-4e71-a427-07c65bf41750-istio-podinfo\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.630665 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.630600 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-workload-socket\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.630665 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.630653 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e28eb1-8011-4e71-a427-07c65bf41750-istio-token\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.630874 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.630749 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-workload-certs\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.630973 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.630950 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-istio-data\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.631040 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.630999 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-credential-socket\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.631040 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.631009 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-workload-socket\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.631162 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.631142 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e28eb1-8011-4e71-a427-07c65bf41750-istiod-ca-cert\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.632900 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.632876 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-istio-envoy\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.633088 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.633071 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e28eb1-8011-4e71-a427-07c65bf41750-istio-podinfo\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.639397 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.639368 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhn9v\" (UniqueName: \"kubernetes.io/projected/99e28eb1-8011-4e71-a427-07c65bf41750-kube-api-access-zhn9v\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.639532 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.639518 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e28eb1-8011-4e71-a427-07c65bf41750-istio-token\") pod \"e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:11.755929 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:11.755828 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:12.034127 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:12.034100 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw"] Sep 05 06:43:12.036221 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:43:12.036190 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e28eb1_8011_4e71_a427_07c65bf41750.slice/crio-1b55be209979703175402ec2a8f4e7cfca75cc99abb4b62dbb394fd7d6e3d9b3 WatchSource:0}: Error finding container 1b55be209979703175402ec2a8f4e7cfca75cc99abb4b62dbb394fd7d6e3d9b3: Status 404 returned error can't find the container with id 1b55be209979703175402ec2a8f4e7cfca75cc99abb4b62dbb394fd7d6e3d9b3 Sep 05 06:43:12.038426 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:12.038373 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:43:12.038526 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:12.038479 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:43:12.038526 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:12.038521 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:43:12.280237 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:12.280195 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" event={"ID":"99e28eb1-8011-4e71-a427-07c65bf41750","Type":"ContainerStarted","Data":"3b0644aa5f335fda3219f8efb3fc2d3bef066d24cd0f8810fe83938e0db83e91"} Sep 05 06:43:12.280483 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:12.280266 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" event={"ID":"99e28eb1-8011-4e71-a427-07c65bf41750","Type":"ContainerStarted","Data":"1b55be209979703175402ec2a8f4e7cfca75cc99abb4b62dbb394fd7d6e3d9b3"} Sep 05 06:43:12.301473 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:12.301352 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" podStartSLOduration=1.301296358 podStartE2EDuration="1.301296358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:43:12.300110674 +0000 UTC m=+1016.684653797" watchObservedRunningTime="2026-09-05 06:43:12.301296358 +0000 UTC m=+1016.685839483" Sep 05 06:43:12.756262 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:12.756222 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:13.392326 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:13.392257 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v"] Sep 05 06:43:13.392586 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:13.392560 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" podUID="81fe750c-eb9c-4be0-9b1c-104d05da6b1d" containerName="istio-proxy" containerID="cri-o://85e342ffee928bbc4d3d5c3262a446b0d6b8665db7a0f36cc6af1921751fa984" gracePeriod=30 Sep 05 06:43:13.737956 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:13.737869 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7080a4-csxh8"] Sep 05 06:43:13.741439 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:13.741410 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7080a4-csxh8" Sep 05 06:43:13.743818 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:13.743791 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 05 06:43:13.743818 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:13.743795 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slfqj\"" Sep 05 06:43:13.750030 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:13.749984 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7080a4-csxh8"] Sep 05 06:43:13.763191 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:13.763159 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:13.862791 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:13.862757 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/836804ad-2afe-4a56-8931-67dfbfc12c24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7080a4-csxh8\" (UID: \"836804ad-2afe-4a56-8931-67dfbfc12c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7080a4-csxh8" Sep 05 06:43:13.964061 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:13.964031 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/836804ad-2afe-4a56-8931-67dfbfc12c24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7080a4-csxh8\" (UID: \"836804ad-2afe-4a56-8931-67dfbfc12c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7080a4-csxh8" Sep 05 06:43:13.964691 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:13.964671 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/836804ad-2afe-4a56-8931-67dfbfc12c24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7080a4-csxh8\" (UID: \"836804ad-2afe-4a56-8931-67dfbfc12c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7080a4-csxh8" Sep 05 06:43:14.053335 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:14.053214 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7080a4-csxh8" Sep 05 06:43:14.294509 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:14.294479 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:14.296091 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:14.296051 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:14.334036 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:14.334000 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7080a4-csxh8"] Sep 05 06:43:14.338134 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:43:14.338103 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod836804ad_2afe_4a56_8931_67dfbfc12c24.slice/crio-3939174289d38f8edf89119fb6259758ac1c9014540dfedbde7623dedb2e28ed WatchSource:0}: Error finding container 3939174289d38f8edf89119fb6259758ac1c9014540dfedbde7623dedb2e28ed: Status 404 returned error can't find the container with id 3939174289d38f8edf89119fb6259758ac1c9014540dfedbde7623dedb2e28ed Sep 05 06:43:15.300698 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:15.300660 2565 generic.go:356] "Generic (PLEG): container finished" podID="836804ad-2afe-4a56-8931-67dfbfc12c24" containerID="744c9af622132a7f3ebe6d751356d5bfd13feb2707b251d091020144140dc079" exitCode=0 Sep 05 06:43:15.301120 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:15.300723 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7080a4-csxh8" event={"ID":"836804ad-2afe-4a56-8931-67dfbfc12c24","Type":"ContainerDied","Data":"744c9af622132a7f3ebe6d751356d5bfd13feb2707b251d091020144140dc079"} Sep 05 06:43:15.301120 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:15.300789 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7080a4-csxh8" event={"ID":"836804ad-2afe-4a56-8931-67dfbfc12c24","Type":"ContainerStarted","Data":"3939174289d38f8edf89119fb6259758ac1c9014540dfedbde7623dedb2e28ed"} Sep 05 06:43:16.449073 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:16.449047 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7080a4-csxh8" Sep 05 06:43:16.589946 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:16.589911 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/836804ad-2afe-4a56-8931-67dfbfc12c24-maas-api-service-ca\") pod \"836804ad-2afe-4a56-8931-67dfbfc12c24\" (UID: \"836804ad-2afe-4a56-8931-67dfbfc12c24\") " Sep 05 06:43:16.590270 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:16.590249 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/836804ad-2afe-4a56-8931-67dfbfc12c24-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "836804ad-2afe-4a56-8931-67dfbfc12c24" (UID: "836804ad-2afe-4a56-8931-67dfbfc12c24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:43:16.691062 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:16.690976 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/836804ad-2afe-4a56-8931-67dfbfc12c24-maas-api-service-ca\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:17.313791 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:17.313760 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7080a4-csxh8" Sep 05 06:43:17.313791 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:17.313786 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7080a4-csxh8" event={"ID":"836804ad-2afe-4a56-8931-67dfbfc12c24","Type":"ContainerDied","Data":"3939174289d38f8edf89119fb6259758ac1c9014540dfedbde7623dedb2e28ed"} Sep 05 06:43:17.314000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:17.313832 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3939174289d38f8edf89119fb6259758ac1c9014540dfedbde7623dedb2e28ed" Sep 05 06:43:18.657422 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.657341 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:43:18.810889 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.810860 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istio-token\") pod \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " Sep 05 06:43:18.810889 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.810895 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istiod-ca-cert\") pod \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " Sep 05 06:43:18.811135 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.810933 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5l9x8\" (UniqueName: \"kubernetes.io/projected/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-kube-api-access-5l9x8\") pod \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " Sep 05 06:43:18.811135 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.810990 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istio-envoy\") pod \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " Sep 05 06:43:18.811135 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.811024 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-workload-certs\") pod \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " Sep 05 06:43:18.811135 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.811076 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istio-data\") pod \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " Sep 05 06:43:18.811135 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.811094 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istio-podinfo\") pod \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " Sep 05 06:43:18.811419 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.811149 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-workload-socket\") pod \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " Sep 05 06:43:18.811419 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.811202 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-credential-socket\") pod \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\" (UID: \"81fe750c-eb9c-4be0-9b1c-104d05da6b1d\") " Sep 05 06:43:18.811520 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.811411 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istio-data" (OuterVolumeSpecName: "istio-data") pod "81fe750c-eb9c-4be0-9b1c-104d05da6b1d" (UID: "81fe750c-eb9c-4be0-9b1c-104d05da6b1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:43:18.811584 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.811559 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "81fe750c-eb9c-4be0-9b1c-104d05da6b1d" (UID: "81fe750c-eb9c-4be0-9b1c-104d05da6b1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:43:18.811639 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.811591 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istio-data\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:18.811695 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.811674 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "81fe750c-eb9c-4be0-9b1c-104d05da6b1d" (UID: "81fe750c-eb9c-4be0-9b1c-104d05da6b1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:43:18.811906 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.811691 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "81fe750c-eb9c-4be0-9b1c-104d05da6b1d" (UID: "81fe750c-eb9c-4be0-9b1c-104d05da6b1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:43:18.811906 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.811735 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "81fe750c-eb9c-4be0-9b1c-104d05da6b1d" (UID: "81fe750c-eb9c-4be0-9b1c-104d05da6b1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:43:18.813528 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.813496 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-kube-api-access-5l9x8" (OuterVolumeSpecName: "kube-api-access-5l9x8") pod "81fe750c-eb9c-4be0-9b1c-104d05da6b1d" (UID: "81fe750c-eb9c-4be0-9b1c-104d05da6b1d"). InnerVolumeSpecName "kube-api-access-5l9x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:43:18.813639 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.813536 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istio-token" (OuterVolumeSpecName: "istio-token") pod "81fe750c-eb9c-4be0-9b1c-104d05da6b1d" (UID: "81fe750c-eb9c-4be0-9b1c-104d05da6b1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:43:18.813639 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.813533 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "81fe750c-eb9c-4be0-9b1c-104d05da6b1d" (UID: "81fe750c-eb9c-4be0-9b1c-104d05da6b1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:43:18.813639 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.813576 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "81fe750c-eb9c-4be0-9b1c-104d05da6b1d" (UID: "81fe750c-eb9c-4be0-9b1c-104d05da6b1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 05 06:43:18.913118 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.913019 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-credential-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:18.913118 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.913061 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istio-token\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:18.913118 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.913072 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istiod-ca-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:18.913118 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.913080 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l9x8\" (UniqueName: \"kubernetes.io/projected/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-kube-api-access-5l9x8\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:18.913118 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.913089 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istio-envoy\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:18.913118 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.913098 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-workload-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:18.913118 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.913106 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-istio-podinfo\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:18.913118 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:18.913114 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fe750c-eb9c-4be0-9b1c-104d05da6b1d-workload-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:19.325461 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:19.325354 2565 generic.go:356] "Generic (PLEG): container finished" podID="81fe750c-eb9c-4be0-9b1c-104d05da6b1d" containerID="85e342ffee928bbc4d3d5c3262a446b0d6b8665db7a0f36cc6af1921751fa984" exitCode=0 Sep 05 06:43:19.325461 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:19.325412 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" event={"ID":"81fe750c-eb9c-4be0-9b1c-104d05da6b1d","Type":"ContainerDied","Data":"85e342ffee928bbc4d3d5c3262a446b0d6b8665db7a0f36cc6af1921751fa984"} Sep 05 06:43:19.325461 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:19.325425 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" Sep 05 06:43:19.325461 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:19.325456 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v" event={"ID":"81fe750c-eb9c-4be0-9b1c-104d05da6b1d","Type":"ContainerDied","Data":"2c0c739d3db82874bde3a376a7d7e579ea566fd2107fd05e5ca584bec7e9bd92"} Sep 05 06:43:19.325779 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:19.325482 2565 scope.go:117] "RemoveContainer" containerID="85e342ffee928bbc4d3d5c3262a446b0d6b8665db7a0f36cc6af1921751fa984" Sep 05 06:43:19.335073 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:19.335057 2565 scope.go:117] "RemoveContainer" containerID="4c69f78135dd03daa1f8c92a27a9d40febc5266b925d1fb7b76be28677c2f893" Sep 05 06:43:19.343792 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:19.343770 2565 scope.go:117] "RemoveContainer" containerID="85e342ffee928bbc4d3d5c3262a446b0d6b8665db7a0f36cc6af1921751fa984" Sep 05 06:43:19.344095 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:43:19.344071 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85e342ffee928bbc4d3d5c3262a446b0d6b8665db7a0f36cc6af1921751fa984\": container with ID starting with 85e342ffee928bbc4d3d5c3262a446b0d6b8665db7a0f36cc6af1921751fa984 not found: ID does not exist" containerID="85e342ffee928bbc4d3d5c3262a446b0d6b8665db7a0f36cc6af1921751fa984" Sep 05 06:43:19.344164 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:19.344105 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e342ffee928bbc4d3d5c3262a446b0d6b8665db7a0f36cc6af1921751fa984"} err="failed to get container status \"85e342ffee928bbc4d3d5c3262a446b0d6b8665db7a0f36cc6af1921751fa984\": rpc error: code = NotFound desc = could not find container \"85e342ffee928bbc4d3d5c3262a446b0d6b8665db7a0f36cc6af1921751fa984\": container with ID starting with 85e342ffee928bbc4d3d5c3262a446b0d6b8665db7a0f36cc6af1921751fa984 not found: ID does not exist" Sep 05 06:43:19.344164 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:19.344123 2565 scope.go:117] "RemoveContainer" containerID="4c69f78135dd03daa1f8c92a27a9d40febc5266b925d1fb7b76be28677c2f893" Sep 05 06:43:19.344409 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:43:19.344391 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c69f78135dd03daa1f8c92a27a9d40febc5266b925d1fb7b76be28677c2f893\": container with ID starting with 4c69f78135dd03daa1f8c92a27a9d40febc5266b925d1fb7b76be28677c2f893 not found: ID does not exist" containerID="4c69f78135dd03daa1f8c92a27a9d40febc5266b925d1fb7b76be28677c2f893" Sep 05 06:43:19.344447 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:19.344416 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c69f78135dd03daa1f8c92a27a9d40febc5266b925d1fb7b76be28677c2f893"} err="failed to get container status \"4c69f78135dd03daa1f8c92a27a9d40febc5266b925d1fb7b76be28677c2f893\": rpc error: code = NotFound desc = could not find container \"4c69f78135dd03daa1f8c92a27a9d40febc5266b925d1fb7b76be28677c2f893\": container with ID starting with 4c69f78135dd03daa1f8c92a27a9d40febc5266b925d1fb7b76be28677c2f893 not found: ID does not exist" Sep 05 06:43:19.350207 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:19.350182 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v"] Sep 05 06:43:19.355656 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:19.355637 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-45e697-openshift-default-66c5649466-g4b2v"] Sep 05 06:43:20.167763 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:20.167726 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81fe750c-eb9c-4be0-9b1c-104d05da6b1d" path="/var/lib/kubelet/pods/81fe750c-eb9c-4be0-9b1c-104d05da6b1d/volumes" Sep 05 06:43:24.626084 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:24.626054 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n"] Sep 05 06:43:24.626606 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:24.626259 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n" podUID="7e61038e-bc4e-42a7-bdb1-4c18fc425388" containerName="payload-processing" containerID="cri-o://ae18dedf3d7c1d27b14e20b0a834a07dc7aaf69cbd983ee9027b556aa249c0a0" gracePeriod=30 Sep 05 06:43:24.636384 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:24.636355 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl"] Sep 05 06:43:24.636900 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:24.636812 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl" podUID="02147d7b-95d4-4690-8e7e-edd85db122e3" containerName="payload-pre-processing" containerID="cri-o://2bd8bee13de2cb8473b08b98342b414000ff2e765b592c5030fbfe7c43b50af9" gracePeriod=30 Sep 05 06:43:24.808638 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:24.808598 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7080a4-csxh8"] Sep 05 06:43:24.811682 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:24.811650 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7080a4-csxh8"] Sep 05 06:43:24.981534 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:24.981170 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl" Sep 05 06:43:25.001536 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.001504 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n" Sep 05 06:43:25.070800 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.070770 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2wsd\" (UniqueName: \"kubernetes.io/projected/7e61038e-bc4e-42a7-bdb1-4c18fc425388-kube-api-access-j2wsd\") pod \"7e61038e-bc4e-42a7-bdb1-4c18fc425388\" (UID: \"7e61038e-bc4e-42a7-bdb1-4c18fc425388\") " Sep 05 06:43:25.070978 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.070850 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02147d7b-95d4-4690-8e7e-edd85db122e3-plugins-config-volume\") pod \"02147d7b-95d4-4690-8e7e-edd85db122e3\" (UID: \"02147d7b-95d4-4690-8e7e-edd85db122e3\") " Sep 05 06:43:25.070978 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.070876 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgc9s\" (UniqueName: \"kubernetes.io/projected/02147d7b-95d4-4690-8e7e-edd85db122e3-kube-api-access-xgc9s\") pod \"02147d7b-95d4-4690-8e7e-edd85db122e3\" (UID: \"02147d7b-95d4-4690-8e7e-edd85db122e3\") " Sep 05 06:43:25.070978 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.070954 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e61038e-bc4e-42a7-bdb1-4c18fc425388-plugins-config-volume\") pod \"7e61038e-bc4e-42a7-bdb1-4c18fc425388\" (UID: \"7e61038e-bc4e-42a7-bdb1-4c18fc425388\") " Sep 05 06:43:25.071241 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.071213 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02147d7b-95d4-4690-8e7e-edd85db122e3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "02147d7b-95d4-4690-8e7e-edd85db122e3" (UID: "02147d7b-95d4-4690-8e7e-edd85db122e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:43:25.071301 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.071281 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e61038e-bc4e-42a7-bdb1-4c18fc425388-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7e61038e-bc4e-42a7-bdb1-4c18fc425388" (UID: "7e61038e-bc4e-42a7-bdb1-4c18fc425388"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:43:25.073147 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.073125 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02147d7b-95d4-4690-8e7e-edd85db122e3-kube-api-access-xgc9s" (OuterVolumeSpecName: "kube-api-access-xgc9s") pod "02147d7b-95d4-4690-8e7e-edd85db122e3" (UID: "02147d7b-95d4-4690-8e7e-edd85db122e3"). InnerVolumeSpecName "kube-api-access-xgc9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:43:25.073203 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.073149 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e61038e-bc4e-42a7-bdb1-4c18fc425388-kube-api-access-j2wsd" (OuterVolumeSpecName: "kube-api-access-j2wsd") pod "7e61038e-bc4e-42a7-bdb1-4c18fc425388" (UID: "7e61038e-bc4e-42a7-bdb1-4c18fc425388"). InnerVolumeSpecName "kube-api-access-j2wsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:43:25.171747 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.171696 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02147d7b-95d4-4690-8e7e-edd85db122e3-plugins-config-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:25.171747 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.171738 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgc9s\" (UniqueName: \"kubernetes.io/projected/02147d7b-95d4-4690-8e7e-edd85db122e3-kube-api-access-xgc9s\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:25.171747 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.171748 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e61038e-bc4e-42a7-bdb1-4c18fc425388-plugins-config-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:25.171747 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.171756 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2wsd\" (UniqueName: \"kubernetes.io/projected/7e61038e-bc4e-42a7-bdb1-4c18fc425388-kube-api-access-j2wsd\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:25.361795 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.361761 2565 generic.go:356] "Generic (PLEG): container finished" podID="02147d7b-95d4-4690-8e7e-edd85db122e3" containerID="2bd8bee13de2cb8473b08b98342b414000ff2e765b592c5030fbfe7c43b50af9" exitCode=0 Sep 05 06:43:25.361971 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.361806 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl" Sep 05 06:43:25.361971 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.361840 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl" event={"ID":"02147d7b-95d4-4690-8e7e-edd85db122e3","Type":"ContainerDied","Data":"2bd8bee13de2cb8473b08b98342b414000ff2e765b592c5030fbfe7c43b50af9"} Sep 05 06:43:25.361971 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.361893 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl" event={"ID":"02147d7b-95d4-4690-8e7e-edd85db122e3","Type":"ContainerDied","Data":"1a56117547709524feb2f84c90ae322c0afc0dd1949748f4976af390e0a3f6b5"} Sep 05 06:43:25.361971 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.361922 2565 scope.go:117] "RemoveContainer" containerID="2bd8bee13de2cb8473b08b98342b414000ff2e765b592c5030fbfe7c43b50af9" Sep 05 06:43:25.363477 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.363353 2565 generic.go:356] "Generic (PLEG): container finished" podID="7e61038e-bc4e-42a7-bdb1-4c18fc425388" containerID="ae18dedf3d7c1d27b14e20b0a834a07dc7aaf69cbd983ee9027b556aa249c0a0" exitCode=0 Sep 05 06:43:25.363477 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.363385 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n" event={"ID":"7e61038e-bc4e-42a7-bdb1-4c18fc425388","Type":"ContainerDied","Data":"ae18dedf3d7c1d27b14e20b0a834a07dc7aaf69cbd983ee9027b556aa249c0a0"} Sep 05 06:43:25.363477 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.363441 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n" event={"ID":"7e61038e-bc4e-42a7-bdb1-4c18fc425388","Type":"ContainerDied","Data":"c0010576e774b7c58522af92a4577c1864a8024b46532c0f74d7f298412c4fbe"} Sep 05 06:43:25.363477 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.363396 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n" Sep 05 06:43:25.372212 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.372195 2565 scope.go:117] "RemoveContainer" containerID="2bd8bee13de2cb8473b08b98342b414000ff2e765b592c5030fbfe7c43b50af9" Sep 05 06:43:25.372540 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:43:25.372522 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bd8bee13de2cb8473b08b98342b414000ff2e765b592c5030fbfe7c43b50af9\": container with ID starting with 2bd8bee13de2cb8473b08b98342b414000ff2e765b592c5030fbfe7c43b50af9 not found: ID does not exist" containerID="2bd8bee13de2cb8473b08b98342b414000ff2e765b592c5030fbfe7c43b50af9" Sep 05 06:43:25.372630 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.372547 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bd8bee13de2cb8473b08b98342b414000ff2e765b592c5030fbfe7c43b50af9"} err="failed to get container status \"2bd8bee13de2cb8473b08b98342b414000ff2e765b592c5030fbfe7c43b50af9\": rpc error: code = NotFound desc = could not find container \"2bd8bee13de2cb8473b08b98342b414000ff2e765b592c5030fbfe7c43b50af9\": container with ID starting with 2bd8bee13de2cb8473b08b98342b414000ff2e765b592c5030fbfe7c43b50af9 not found: ID does not exist" Sep 05 06:43:25.372630 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.372568 2565 scope.go:117] "RemoveContainer" containerID="ae18dedf3d7c1d27b14e20b0a834a07dc7aaf69cbd983ee9027b556aa249c0a0" Sep 05 06:43:25.390940 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.390900 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl"] Sep 05 06:43:25.400547 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.398706 2565 scope.go:117] "RemoveContainer" containerID="ae18dedf3d7c1d27b14e20b0a834a07dc7aaf69cbd983ee9027b556aa249c0a0" Sep 05 06:43:25.400547 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:43:25.399106 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae18dedf3d7c1d27b14e20b0a834a07dc7aaf69cbd983ee9027b556aa249c0a0\": container with ID starting with ae18dedf3d7c1d27b14e20b0a834a07dc7aaf69cbd983ee9027b556aa249c0a0 not found: ID does not exist" containerID="ae18dedf3d7c1d27b14e20b0a834a07dc7aaf69cbd983ee9027b556aa249c0a0" Sep 05 06:43:25.400547 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.399145 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae18dedf3d7c1d27b14e20b0a834a07dc7aaf69cbd983ee9027b556aa249c0a0"} err="failed to get container status \"ae18dedf3d7c1d27b14e20b0a834a07dc7aaf69cbd983ee9027b556aa249c0a0\": rpc error: code = NotFound desc = could not find container \"ae18dedf3d7c1d27b14e20b0a834a07dc7aaf69cbd983ee9027b556aa249c0a0\": container with ID starting with ae18dedf3d7c1d27b14e20b0a834a07dc7aaf69cbd983ee9027b556aa249c0a0 not found: ID does not exist" Sep 05 06:43:25.401973 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.401948 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7080a4-f5cfbb45d-rd7tl"] Sep 05 06:43:25.407223 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.407197 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n"] Sep 05 06:43:25.413460 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:25.413434 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7080a4-77486d5cf-fqx6n"] Sep 05 06:43:26.166440 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:26.166406 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02147d7b-95d4-4690-8e7e-edd85db122e3" path="/var/lib/kubelet/pods/02147d7b-95d4-4690-8e7e-edd85db122e3/volumes" Sep 05 06:43:26.166804 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:26.166783 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e61038e-bc4e-42a7-bdb1-4c18fc425388" path="/var/lib/kubelet/pods/7e61038e-bc4e-42a7-bdb1-4c18fc425388/volumes" Sep 05 06:43:26.167047 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:26.167034 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="836804ad-2afe-4a56-8931-67dfbfc12c24" path="/var/lib/kubelet/pods/836804ad-2afe-4a56-8931-67dfbfc12c24/volumes" Sep 05 06:43:29.780817 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:29.779922 2565 patch_prober.go:28] interesting pod/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": read tcp 10.133.0.2:54820->10.133.0.50:15021: read: connection reset by peer" start-of-body= Sep 05 06:43:29.780817 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:29.779991 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" podUID="99e28eb1-8011-4e71-a427-07c65bf41750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": read tcp 10.133.0.2:54820->10.133.0.50:15021: read: connection reset by peer" Sep 05 06:43:30.396603 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:30.396570 2565 generic.go:356] "Generic (PLEG): container finished" podID="99e28eb1-8011-4e71-a427-07c65bf41750" containerID="3b0644aa5f335fda3219f8efb3fc2d3bef066d24cd0f8810fe83938e0db83e91" exitCode=0 Sep 05 06:43:30.396777 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:30.396619 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" event={"ID":"99e28eb1-8011-4e71-a427-07c65bf41750","Type":"ContainerDied","Data":"3b0644aa5f335fda3219f8efb3fc2d3bef066d24cd0f8810fe83938e0db83e91"} Sep 05 06:43:30.397017 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:30.397005 2565 scope.go:117] "RemoveContainer" containerID="3b0644aa5f335fda3219f8efb3fc2d3bef066d24cd0f8810fe83938e0db83e91" Sep 05 06:43:30.398060 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:30.398033 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:43:30.398117 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:30.398105 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:43:30.398151 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:30.398142 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:43:31.407472 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:31.407440 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" event={"ID":"99e28eb1-8011-4e71-a427-07c65bf41750","Type":"ContainerStarted","Data":"1ceb0b4dd9b8562d166e59e1eb57a4c0e1c3852798ffa2f48e9f816dfb306752"} Sep 05 06:43:31.667232 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:31.667152 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk"] Sep 05 06:43:31.667444 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:31.667418 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" podUID="77ba3167-1e93-4e94-8e9a-fc8d96997556" containerName="istio-proxy" containerID="cri-o://d641554afba1f7ab537afd5d7fc8fc30090d60d429ef26eb6915da7002ece924" gracePeriod=30 Sep 05 06:43:31.756523 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:31.756480 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:31.756970 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:31.756939 2565 patch_prober.go:28] interesting pod/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 05 06:43:31.757553 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:31.757172 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" podUID="99e28eb1-8011-4e71-a427-07c65bf41750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 05 06:43:32.558091 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.558059 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg"] Sep 05 06:43:32.558483 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.558447 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81fe750c-eb9c-4be0-9b1c-104d05da6b1d" containerName="istio-proxy" Sep 05 06:43:32.558483 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.558459 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="81fe750c-eb9c-4be0-9b1c-104d05da6b1d" containerName="istio-proxy" Sep 05 06:43:32.558483 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.558471 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81fe750c-eb9c-4be0-9b1c-104d05da6b1d" containerName="istio-proxy" Sep 05 06:43:32.558483 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.558478 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="81fe750c-eb9c-4be0-9b1c-104d05da6b1d" containerName="istio-proxy" Sep 05 06:43:32.558639 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.558488 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e61038e-bc4e-42a7-bdb1-4c18fc425388" containerName="payload-processing" Sep 05 06:43:32.558639 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.558496 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e61038e-bc4e-42a7-bdb1-4c18fc425388" containerName="payload-processing" Sep 05 06:43:32.558639 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.558514 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="836804ad-2afe-4a56-8931-67dfbfc12c24" containerName="revoke-keys" Sep 05 06:43:32.558639 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.558521 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="836804ad-2afe-4a56-8931-67dfbfc12c24" containerName="revoke-keys" Sep 05 06:43:32.558639 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.558538 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02147d7b-95d4-4690-8e7e-edd85db122e3" containerName="payload-pre-processing" Sep 05 06:43:32.558639 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.558543 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="02147d7b-95d4-4690-8e7e-edd85db122e3" containerName="payload-pre-processing" Sep 05 06:43:32.558639 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.558622 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="02147d7b-95d4-4690-8e7e-edd85db122e3" containerName="payload-pre-processing" Sep 05 06:43:32.558639 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.558637 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="81fe750c-eb9c-4be0-9b1c-104d05da6b1d" containerName="istio-proxy" Sep 05 06:43:32.558994 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.558648 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="836804ad-2afe-4a56-8931-67dfbfc12c24" containerName="revoke-keys" Sep 05 06:43:32.558994 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.558658 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="81fe750c-eb9c-4be0-9b1c-104d05da6b1d" containerName="istio-proxy" Sep 05 06:43:32.558994 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.558665 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e61038e-bc4e-42a7-bdb1-4c18fc425388" containerName="payload-processing" Sep 05 06:43:32.561953 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.561933 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.564161 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.564140 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-183ae6-openshift-default-dockercfg-tkvsv\"" Sep 05 06:43:32.582822 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.576138 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg"] Sep 05 06:43:32.651930 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.651890 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7r5s\" (UniqueName: \"kubernetes.io/projected/b89fc0ad-b6f9-4cdd-8786-24883ebac401-kube-api-access-c7r5s\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.652108 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.651949 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istio-data\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.652108 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.652021 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istio-token\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.652108 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.652051 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istio-envoy\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.652108 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.652089 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istio-podinfo\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.652338 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.652121 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istiod-ca-cert\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.652338 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.652146 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-workload-certs\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.652338 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.652201 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-credential-socket\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.652338 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.652235 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-workload-socket\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.753418 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.753364 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7r5s\" (UniqueName: \"kubernetes.io/projected/b89fc0ad-b6f9-4cdd-8786-24883ebac401-kube-api-access-c7r5s\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.753418 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.753424 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istio-data\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.753724 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.753473 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istio-token\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.753724 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.753499 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istio-envoy\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.753724 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.753536 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istio-podinfo\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.753724 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.753565 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istiod-ca-cert\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.753724 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.753589 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-workload-certs\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.753724 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.753618 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-credential-socket\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.753724 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.753649 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-workload-socket\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.754156 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.753866 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istio-data\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.754156 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.753966 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-workload-certs\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.754156 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.754048 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-credential-socket\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.754156 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.754057 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-workload-socket\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.754556 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.754277 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istiod-ca-cert\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.756006 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.755985 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istio-envoy\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.756159 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.756132 2565 patch_prober.go:28] interesting pod/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 05 06:43:32.756272 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.756189 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" podUID="99e28eb1-8011-4e71-a427-07c65bf41750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 05 06:43:32.756437 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.756416 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istio-podinfo\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.761783 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.761757 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istio-token\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.761873 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.761765 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7r5s\" (UniqueName: \"kubernetes.io/projected/b89fc0ad-b6f9-4cdd-8786-24883ebac401-kube-api-access-c7r5s\") pod \"e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:32.873165 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:32.873122 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:33.168865 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:33.168825 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg"] Sep 05 06:43:33.171010 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:43:33.170981 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb89fc0ad_b6f9_4cdd_8786_24883ebac401.slice/crio-d83d63c54af176e6ab4739a2412dacf6a842be1ac1429814452701bd7f54c58a WatchSource:0}: Error finding container d83d63c54af176e6ab4739a2412dacf6a842be1ac1429814452701bd7f54c58a: Status 404 returned error can't find the container with id d83d63c54af176e6ab4739a2412dacf6a842be1ac1429814452701bd7f54c58a Sep 05 06:43:33.173554 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:33.173517 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:43:33.173642 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:33.173601 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:43:33.173681 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:33.173639 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:43:33.431004 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:33.430908 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" event={"ID":"b89fc0ad-b6f9-4cdd-8786-24883ebac401","Type":"ContainerStarted","Data":"7f27a85910310ecc02b034d4a31bea15c2696681fa809759c7c13f8167a5309a"} Sep 05 06:43:33.431283 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:33.431266 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" event={"ID":"b89fc0ad-b6f9-4cdd-8786-24883ebac401","Type":"ContainerStarted","Data":"d83d63c54af176e6ab4739a2412dacf6a842be1ac1429814452701bd7f54c58a"} Sep 05 06:43:33.451884 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:33.451831 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" podStartSLOduration=1.451814207 podStartE2EDuration="1.451814207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:43:33.450508506 +0000 UTC m=+1037.835051629" watchObservedRunningTime="2026-09-05 06:43:33.451814207 +0000 UTC m=+1037.836357331" Sep 05 06:43:33.757041 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:33.756954 2565 patch_prober.go:28] interesting pod/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 05 06:43:33.757041 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:33.757017 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" podUID="99e28eb1-8011-4e71-a427-07c65bf41750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 05 06:43:33.874141 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:33.874106 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:34.756626 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:34.756588 2565 patch_prober.go:28] interesting pod/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 05 06:43:34.756808 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:34.756646 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" podUID="99e28eb1-8011-4e71-a427-07c65bf41750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 05 06:43:34.887559 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:34.887518 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:35.442416 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:35.442379 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:35.443302 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:35.443281 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:35.756983 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:35.756894 2565 patch_prober.go:28] interesting pod/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 05 06:43:35.756983 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:35.756951 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" podUID="99e28eb1-8011-4e71-a427-07c65bf41750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 05 06:43:36.759635 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:36.759608 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:36.760104 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:36.760091 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:36.760769 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:36.760750 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:43:36.925724 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:36.925701 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:43:37.002652 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.002569 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-workload-socket\") pod \"77ba3167-1e93-4e94-8e9a-fc8d96997556\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " Sep 05 06:43:37.002652 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.002619 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ba3167-1e93-4e94-8e9a-fc8d96997556-istiod-ca-cert\") pod \"77ba3167-1e93-4e94-8e9a-fc8d96997556\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " Sep 05 06:43:37.002652 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.002639 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-workload-certs\") pod \"77ba3167-1e93-4e94-8e9a-fc8d96997556\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " Sep 05 06:43:37.002922 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.002678 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-credential-socket\") pod \"77ba3167-1e93-4e94-8e9a-fc8d96997556\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " Sep 05 06:43:37.002922 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.002697 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-istio-data\") pod \"77ba3167-1e93-4e94-8e9a-fc8d96997556\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " Sep 05 06:43:37.002922 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.002805 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-istio-envoy\") pod \"77ba3167-1e93-4e94-8e9a-fc8d96997556\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " Sep 05 06:43:37.002922 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.002859 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "77ba3167-1e93-4e94-8e9a-fc8d96997556" (UID: "77ba3167-1e93-4e94-8e9a-fc8d96997556"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:43:37.002922 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.002869 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ba3167-1e93-4e94-8e9a-fc8d96997556-istio-token\") pod \"77ba3167-1e93-4e94-8e9a-fc8d96997556\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " Sep 05 06:43:37.003146 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.002938 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fx8k2\" (UniqueName: \"kubernetes.io/projected/77ba3167-1e93-4e94-8e9a-fc8d96997556-kube-api-access-fx8k2\") pod \"77ba3167-1e93-4e94-8e9a-fc8d96997556\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " Sep 05 06:43:37.003146 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.002980 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ba3167-1e93-4e94-8e9a-fc8d96997556-istio-podinfo\") pod \"77ba3167-1e93-4e94-8e9a-fc8d96997556\" (UID: \"77ba3167-1e93-4e94-8e9a-fc8d96997556\") " Sep 05 06:43:37.003146 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.002997 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-istio-data" (OuterVolumeSpecName: "istio-data") pod "77ba3167-1e93-4e94-8e9a-fc8d96997556" (UID: "77ba3167-1e93-4e94-8e9a-fc8d96997556"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:43:37.003146 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.003012 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "77ba3167-1e93-4e94-8e9a-fc8d96997556" (UID: "77ba3167-1e93-4e94-8e9a-fc8d96997556"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:43:37.003146 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.003027 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "77ba3167-1e93-4e94-8e9a-fc8d96997556" (UID: "77ba3167-1e93-4e94-8e9a-fc8d96997556"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:43:37.003146 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.003089 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77ba3167-1e93-4e94-8e9a-fc8d96997556-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "77ba3167-1e93-4e94-8e9a-fc8d96997556" (UID: "77ba3167-1e93-4e94-8e9a-fc8d96997556"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:43:37.003439 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.003295 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-workload-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:37.003439 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.003329 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ba3167-1e93-4e94-8e9a-fc8d96997556-istiod-ca-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:37.003439 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.003340 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-workload-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:37.003439 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.003349 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-credential-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:37.003439 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.003358 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-istio-data\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:37.005406 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.005368 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ba3167-1e93-4e94-8e9a-fc8d96997556-istio-token" (OuterVolumeSpecName: "istio-token") pod "77ba3167-1e93-4e94-8e9a-fc8d96997556" (UID: "77ba3167-1e93-4e94-8e9a-fc8d96997556"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:43:37.005528 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.005387 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ba3167-1e93-4e94-8e9a-fc8d96997556-kube-api-access-fx8k2" (OuterVolumeSpecName: "kube-api-access-fx8k2") pod "77ba3167-1e93-4e94-8e9a-fc8d96997556" (UID: "77ba3167-1e93-4e94-8e9a-fc8d96997556"). InnerVolumeSpecName "kube-api-access-fx8k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:43:37.005704 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.005682 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "77ba3167-1e93-4e94-8e9a-fc8d96997556" (UID: "77ba3167-1e93-4e94-8e9a-fc8d96997556"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:43:37.005800 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.005757 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77ba3167-1e93-4e94-8e9a-fc8d96997556-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "77ba3167-1e93-4e94-8e9a-fc8d96997556" (UID: "77ba3167-1e93-4e94-8e9a-fc8d96997556"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 05 06:43:37.104516 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.104478 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ba3167-1e93-4e94-8e9a-fc8d96997556-istio-envoy\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:37.104516 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.104510 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ba3167-1e93-4e94-8e9a-fc8d96997556-istio-token\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:37.104516 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.104521 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx8k2\" (UniqueName: \"kubernetes.io/projected/77ba3167-1e93-4e94-8e9a-fc8d96997556-kube-api-access-fx8k2\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:37.104757 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.104530 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ba3167-1e93-4e94-8e9a-fc8d96997556-istio-podinfo\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:37.455931 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.455895 2565 generic.go:356] "Generic (PLEG): container finished" podID="77ba3167-1e93-4e94-8e9a-fc8d96997556" containerID="d641554afba1f7ab537afd5d7fc8fc30090d60d429ef26eb6915da7002ece924" exitCode=0 Sep 05 06:43:37.456137 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.455935 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" event={"ID":"77ba3167-1e93-4e94-8e9a-fc8d96997556","Type":"ContainerDied","Data":"d641554afba1f7ab537afd5d7fc8fc30090d60d429ef26eb6915da7002ece924"} Sep 05 06:43:37.456137 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.455963 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" Sep 05 06:43:37.456137 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.456006 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk" event={"ID":"77ba3167-1e93-4e94-8e9a-fc8d96997556","Type":"ContainerDied","Data":"a06f0c1e82e775d03edf3633a047f7556bfb9df475326519aea114fda0f0a748"} Sep 05 06:43:37.456137 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.456036 2565 scope.go:117] "RemoveContainer" containerID="d641554afba1f7ab537afd5d7fc8fc30090d60d429ef26eb6915da7002ece924" Sep 05 06:43:37.472503 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.472476 2565 scope.go:117] "RemoveContainer" containerID="1210c2fc59f60dcb17380ad0abcca521d57b13a7fb102f63363c79fb44770cf8" Sep 05 06:43:37.481076 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.481050 2565 scope.go:117] "RemoveContainer" containerID="d641554afba1f7ab537afd5d7fc8fc30090d60d429ef26eb6915da7002ece924" Sep 05 06:43:37.481366 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:43:37.481344 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d641554afba1f7ab537afd5d7fc8fc30090d60d429ef26eb6915da7002ece924\": container with ID starting with d641554afba1f7ab537afd5d7fc8fc30090d60d429ef26eb6915da7002ece924 not found: ID does not exist" containerID="d641554afba1f7ab537afd5d7fc8fc30090d60d429ef26eb6915da7002ece924" Sep 05 06:43:37.481427 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.481376 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d641554afba1f7ab537afd5d7fc8fc30090d60d429ef26eb6915da7002ece924"} err="failed to get container status \"d641554afba1f7ab537afd5d7fc8fc30090d60d429ef26eb6915da7002ece924\": rpc error: code = NotFound desc = could not find container \"d641554afba1f7ab537afd5d7fc8fc30090d60d429ef26eb6915da7002ece924\": container with ID starting with d641554afba1f7ab537afd5d7fc8fc30090d60d429ef26eb6915da7002ece924 not found: ID does not exist" Sep 05 06:43:37.481427 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.481394 2565 scope.go:117] "RemoveContainer" containerID="1210c2fc59f60dcb17380ad0abcca521d57b13a7fb102f63363c79fb44770cf8" Sep 05 06:43:37.481678 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:43:37.481659 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1210c2fc59f60dcb17380ad0abcca521d57b13a7fb102f63363c79fb44770cf8\": container with ID starting with 1210c2fc59f60dcb17380ad0abcca521d57b13a7fb102f63363c79fb44770cf8 not found: ID does not exist" containerID="1210c2fc59f60dcb17380ad0abcca521d57b13a7fb102f63363c79fb44770cf8" Sep 05 06:43:37.481732 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.481681 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1210c2fc59f60dcb17380ad0abcca521d57b13a7fb102f63363c79fb44770cf8"} err="failed to get container status \"1210c2fc59f60dcb17380ad0abcca521d57b13a7fb102f63363c79fb44770cf8\": rpc error: code = NotFound desc = could not find container \"1210c2fc59f60dcb17380ad0abcca521d57b13a7fb102f63363c79fb44770cf8\": container with ID starting with 1210c2fc59f60dcb17380ad0abcca521d57b13a7fb102f63363c79fb44770cf8 not found: ID does not exist" Sep 05 06:43:37.486318 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.486280 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk"] Sep 05 06:43:37.490652 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:37.490630 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7080a4-openshift-default-699b5dc7c6-vbkfk"] Sep 05 06:43:38.166424 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.166394 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77ba3167-1e93-4e94-8e9a-fc8d96997556" path="/var/lib/kubelet/pods/77ba3167-1e93-4e94-8e9a-fc8d96997556/volumes" Sep 05 06:43:38.175649 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.175622 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd"] Sep 05 06:43:38.176037 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.176020 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77ba3167-1e93-4e94-8e9a-fc8d96997556" containerName="istio-proxy" Sep 05 06:43:38.176110 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.176040 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="77ba3167-1e93-4e94-8e9a-fc8d96997556" containerName="istio-proxy" Sep 05 06:43:38.176159 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.176153 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="77ba3167-1e93-4e94-8e9a-fc8d96997556" containerName="istio-proxy" Sep 05 06:43:38.180648 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.180627 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd" Sep 05 06:43:38.183134 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.183116 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-183ae6\"" Sep 05 06:43:38.183339 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.183322 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-183ae6-dockercfg-gnvsw\"" Sep 05 06:43:38.189761 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.189738 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd"] Sep 05 06:43:38.234670 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.234637 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq"] Sep 05 06:43:38.235029 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.235017 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77ba3167-1e93-4e94-8e9a-fc8d96997556" containerName="istio-proxy" Sep 05 06:43:38.235087 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.235031 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="77ba3167-1e93-4e94-8e9a-fc8d96997556" containerName="istio-proxy" Sep 05 06:43:38.235125 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.235111 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="77ba3167-1e93-4e94-8e9a-fc8d96997556" containerName="istio-proxy" Sep 05 06:43:38.238204 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.238187 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq" Sep 05 06:43:38.248595 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.248573 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq"] Sep 05 06:43:38.317103 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.317063 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b69b6519-cb9a-482a-8fbd-902283fe6a42-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd\" (UID: \"b69b6519-cb9a-482a-8fbd-902283fe6a42\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd" Sep 05 06:43:38.317263 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.317121 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250ab2e7-8101-463f-8dee-591d425a1469-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq\" (UID: \"250ab2e7-8101-463f-8dee-591d425a1469\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq" Sep 05 06:43:38.317263 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.317188 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fmkg\" (UniqueName: \"kubernetes.io/projected/250ab2e7-8101-463f-8dee-591d425a1469-kube-api-access-5fmkg\") pod \"payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq\" (UID: \"250ab2e7-8101-463f-8dee-591d425a1469\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq" Sep 05 06:43:38.317263 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.317219 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb9bs\" (UniqueName: \"kubernetes.io/projected/b69b6519-cb9a-482a-8fbd-902283fe6a42-kube-api-access-pb9bs\") pod \"payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd\" (UID: \"b69b6519-cb9a-482a-8fbd-902283fe6a42\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd" Sep 05 06:43:38.418218 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.418125 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b69b6519-cb9a-482a-8fbd-902283fe6a42-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd\" (UID: \"b69b6519-cb9a-482a-8fbd-902283fe6a42\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd" Sep 05 06:43:38.418218 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.418178 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250ab2e7-8101-463f-8dee-591d425a1469-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq\" (UID: \"250ab2e7-8101-463f-8dee-591d425a1469\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq" Sep 05 06:43:38.418218 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.418209 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fmkg\" (UniqueName: \"kubernetes.io/projected/250ab2e7-8101-463f-8dee-591d425a1469-kube-api-access-5fmkg\") pod \"payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq\" (UID: \"250ab2e7-8101-463f-8dee-591d425a1469\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq" Sep 05 06:43:38.418474 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.418243 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb9bs\" (UniqueName: \"kubernetes.io/projected/b69b6519-cb9a-482a-8fbd-902283fe6a42-kube-api-access-pb9bs\") pod \"payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd\" (UID: \"b69b6519-cb9a-482a-8fbd-902283fe6a42\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd" Sep 05 06:43:38.419095 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.419070 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250ab2e7-8101-463f-8dee-591d425a1469-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq\" (UID: \"250ab2e7-8101-463f-8dee-591d425a1469\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq" Sep 05 06:43:38.419137 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.419070 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b69b6519-cb9a-482a-8fbd-902283fe6a42-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd\" (UID: \"b69b6519-cb9a-482a-8fbd-902283fe6a42\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd" Sep 05 06:43:38.428343 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.428272 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb9bs\" (UniqueName: \"kubernetes.io/projected/b69b6519-cb9a-482a-8fbd-902283fe6a42-kube-api-access-pb9bs\") pod \"payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd\" (UID: \"b69b6519-cb9a-482a-8fbd-902283fe6a42\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd" Sep 05 06:43:38.428540 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.428352 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fmkg\" (UniqueName: \"kubernetes.io/projected/250ab2e7-8101-463f-8dee-591d425a1469-kube-api-access-5fmkg\") pod \"payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq\" (UID: \"250ab2e7-8101-463f-8dee-591d425a1469\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq" Sep 05 06:43:38.491664 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.491632 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd" Sep 05 06:43:38.549726 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.549692 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq" Sep 05 06:43:38.773973 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.773893 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd"] Sep 05 06:43:38.777117 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:43:38.777088 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69b6519_cb9a_482a_8fbd_902283fe6a42.slice/crio-af964c2f79e6363e8b8a335e16fd9db398566f6770b41eaf81d38be1a3ee298e WatchSource:0}: Error finding container af964c2f79e6363e8b8a335e16fd9db398566f6770b41eaf81d38be1a3ee298e: Status 404 returned error can't find the container with id af964c2f79e6363e8b8a335e16fd9db398566f6770b41eaf81d38be1a3ee298e Sep 05 06:43:38.840194 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:38.840093 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq"] Sep 05 06:43:38.842360 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:43:38.842333 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250ab2e7_8101_463f_8dee_591d425a1469.slice/crio-5536e138c4858df133f4cc8d8e0d2c084f1789f919a96df8a3db6c4c2cd473f2 WatchSource:0}: Error finding container 5536e138c4858df133f4cc8d8e0d2c084f1789f919a96df8a3db6c4c2cd473f2: Status 404 returned error can't find the container with id 5536e138c4858df133f4cc8d8e0d2c084f1789f919a96df8a3db6c4c2cd473f2 Sep 05 06:43:39.470843 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:39.470804 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd" event={"ID":"b69b6519-cb9a-482a-8fbd-902283fe6a42","Type":"ContainerStarted","Data":"dfdc139d06f8ad8012a5b58af70925d6d9361a563f61ba26f10205e5a99dac52"} Sep 05 06:43:39.471290 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:39.470871 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd" event={"ID":"b69b6519-cb9a-482a-8fbd-902283fe6a42","Type":"ContainerStarted","Data":"af964c2f79e6363e8b8a335e16fd9db398566f6770b41eaf81d38be1a3ee298e"} Sep 05 06:43:39.471290 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:39.470904 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd" Sep 05 06:43:39.472426 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:39.472393 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq" event={"ID":"250ab2e7-8101-463f-8dee-591d425a1469","Type":"ContainerStarted","Data":"c572c31c3218eee82b74935c2f82d12209dcc12b983024d175c6e5d45f502302"} Sep 05 06:43:39.472558 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:39.472444 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq" event={"ID":"250ab2e7-8101-463f-8dee-591d425a1469","Type":"ContainerStarted","Data":"5536e138c4858df133f4cc8d8e0d2c084f1789f919a96df8a3db6c4c2cd473f2"} Sep 05 06:43:39.472558 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:39.472483 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq" Sep 05 06:43:39.489392 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:39.489332 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd" podStartSLOduration=1.489294062 podStartE2EDuration="1.489294062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:43:39.487292218 +0000 UTC m=+1043.871835364" watchObservedRunningTime="2026-09-05 06:43:39.489294062 +0000 UTC m=+1043.873837187" Sep 05 06:43:39.508805 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:39.508752 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq" podStartSLOduration=1.508736366 podStartE2EDuration="1.508736366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:43:39.507050733 +0000 UTC m=+1043.891593942" watchObservedRunningTime="2026-09-05 06:43:39.508736366 +0000 UTC m=+1043.893279489" Sep 05 06:43:47.021694 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:47.021658 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-810a2bb6-4728t"] Sep 05 06:43:47.025506 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:47.025488 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-810a2bb6-4728t" Sep 05 06:43:47.028115 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:47.028088 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 05 06:43:47.028242 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:47.028098 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slfqj\"" Sep 05 06:43:47.035665 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:47.035293 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-810a2bb6-4728t"] Sep 05 06:43:47.122842 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:47.122811 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/579bb08b-8f4a-4731-9a08-9a189e7c0778-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-810a2bb6-4728t\" (UID: \"579bb08b-8f4a-4731-9a08-9a189e7c0778\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-810a2bb6-4728t" Sep 05 06:43:47.223964 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:47.223926 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/579bb08b-8f4a-4731-9a08-9a189e7c0778-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-810a2bb6-4728t\" (UID: \"579bb08b-8f4a-4731-9a08-9a189e7c0778\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-810a2bb6-4728t" Sep 05 06:43:47.224669 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:47.224641 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/579bb08b-8f4a-4731-9a08-9a189e7c0778-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-810a2bb6-4728t\" (UID: \"579bb08b-8f4a-4731-9a08-9a189e7c0778\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-810a2bb6-4728t" Sep 05 06:43:47.339756 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:47.339648 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-810a2bb6-4728t" Sep 05 06:43:47.621981 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:47.621954 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-810a2bb6-4728t"] Sep 05 06:43:47.624122 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:43:47.624077 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod579bb08b_8f4a_4731_9a08_9a189e7c0778.slice/crio-f2ab1353cbbd6cf0388e377aa75e9967b298a230b18093d2950a9470d2748931 WatchSource:0}: Error finding container f2ab1353cbbd6cf0388e377aa75e9967b298a230b18093d2950a9470d2748931: Status 404 returned error can't find the container with id f2ab1353cbbd6cf0388e377aa75e9967b298a230b18093d2950a9470d2748931 Sep 05 06:43:48.528991 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:48.528899 2565 generic.go:356] "Generic (PLEG): container finished" podID="579bb08b-8f4a-4731-9a08-9a189e7c0778" containerID="b316fb9b7abb74fa5b4d94c2619e279de0a8ba4593168f3da12c51479683cbef" exitCode=0 Sep 05 06:43:48.528991 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:48.528953 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-810a2bb6-4728t" event={"ID":"579bb08b-8f4a-4731-9a08-9a189e7c0778","Type":"ContainerDied","Data":"b316fb9b7abb74fa5b4d94c2619e279de0a8ba4593168f3da12c51479683cbef"} Sep 05 06:43:48.528991 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:48.528993 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-810a2bb6-4728t" event={"ID":"579bb08b-8f4a-4731-9a08-9a189e7c0778","Type":"ContainerStarted","Data":"f2ab1353cbbd6cf0388e377aa75e9967b298a230b18093d2950a9470d2748931"} Sep 05 06:43:49.672508 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:49.672482 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-810a2bb6-4728t" Sep 05 06:43:49.850987 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:49.850961 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/579bb08b-8f4a-4731-9a08-9a189e7c0778-maas-api-service-ca\") pod \"579bb08b-8f4a-4731-9a08-9a189e7c0778\" (UID: \"579bb08b-8f4a-4731-9a08-9a189e7c0778\") " Sep 05 06:43:49.851340 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:49.851299 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/579bb08b-8f4a-4731-9a08-9a189e7c0778-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "579bb08b-8f4a-4731-9a08-9a189e7c0778" (UID: "579bb08b-8f4a-4731-9a08-9a189e7c0778"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:43:49.953277 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:49.953234 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/579bb08b-8f4a-4731-9a08-9a189e7c0778-maas-api-service-ca\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:43:50.444808 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:50.444720 2565 patch_prober.go:28] interesting pod/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 05 06:43:50.444808 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:50.444779 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" podUID="b89fc0ad-b6f9-4cdd-8786-24883ebac401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 05 06:43:50.479860 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:50.479832 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq" Sep 05 06:43:50.480018 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:50.479898 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd" Sep 05 06:43:50.542178 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:50.542145 2565 generic.go:356] "Generic (PLEG): container finished" podID="b89fc0ad-b6f9-4cdd-8786-24883ebac401" containerID="7f27a85910310ecc02b034d4a31bea15c2696681fa809759c7c13f8167a5309a" exitCode=0 Sep 05 06:43:50.542370 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:50.542225 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" event={"ID":"b89fc0ad-b6f9-4cdd-8786-24883ebac401","Type":"ContainerDied","Data":"7f27a85910310ecc02b034d4a31bea15c2696681fa809759c7c13f8167a5309a"} Sep 05 06:43:50.542904 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:50.542879 2565 scope.go:117] "RemoveContainer" containerID="7f27a85910310ecc02b034d4a31bea15c2696681fa809759c7c13f8167a5309a" Sep 05 06:43:50.543846 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:50.543696 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-810a2bb6-4728t" Sep 05 06:43:50.543846 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:50.543738 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-810a2bb6-4728t" event={"ID":"579bb08b-8f4a-4731-9a08-9a189e7c0778","Type":"ContainerDied","Data":"f2ab1353cbbd6cf0388e377aa75e9967b298a230b18093d2950a9470d2748931"} Sep 05 06:43:50.543846 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:50.543786 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2ab1353cbbd6cf0388e377aa75e9967b298a230b18093d2950a9470d2748931" Sep 05 06:43:50.544124 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:50.544098 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:43:50.544200 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:50.544182 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:43:50.544249 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:50.544215 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:43:51.550753 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:51.550718 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" event={"ID":"b89fc0ad-b6f9-4cdd-8786-24883ebac401","Type":"ContainerStarted","Data":"ef55def92258fdfdfc00e2591cb204b8cd34ab219e27901b177e81a44e9dc896"} Sep 05 06:43:51.874011 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:51.873984 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:51.874404 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:51.874380 2565 patch_prober.go:28] interesting pod/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 05 06:43:51.874513 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:51.874436 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" podUID="b89fc0ad-b6f9-4cdd-8786-24883ebac401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 05 06:43:52.873970 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:52.873938 2565 patch_prober.go:28] interesting pod/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 05 06:43:52.874402 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:52.873993 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" podUID="b89fc0ad-b6f9-4cdd-8786-24883ebac401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 05 06:43:53.874589 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:53.874548 2565 patch_prober.go:28] interesting pod/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 05 06:43:53.875087 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:53.874615 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" podUID="b89fc0ad-b6f9-4cdd-8786-24883ebac401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 05 06:43:54.873729 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:54.873696 2565 patch_prober.go:28] interesting pod/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 05 06:43:54.873940 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:54.873754 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" podUID="b89fc0ad-b6f9-4cdd-8786-24883ebac401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 05 06:43:55.873652 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:55.873616 2565 patch_prober.go:28] interesting pod/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 05 06:43:55.874125 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:55.873676 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" podUID="b89fc0ad-b6f9-4cdd-8786-24883ebac401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 05 06:43:56.878036 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:56.878005 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:56.878430 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:56.878273 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:56.878976 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:56.878961 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:43:58.029066 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:58.029027 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-810a2bb6-4728t"] Sep 05 06:43:58.031991 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:58.031960 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-810a2bb6-4728t"] Sep 05 06:43:58.166762 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:43:58.166732 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="579bb08b-8f4a-4731-9a08-9a189e7c0778" path="/var/lib/kubelet/pods/579bb08b-8f4a-4731-9a08-9a189e7c0778/volumes" Sep 05 06:44:01.014336 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.014279 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx"] Sep 05 06:44:01.014820 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.014803 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="579bb08b-8f4a-4731-9a08-9a189e7c0778" containerName="revoke-keys" Sep 05 06:44:01.014865 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.014822 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="579bb08b-8f4a-4731-9a08-9a189e7c0778" containerName="revoke-keys" Sep 05 06:44:01.014903 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.014894 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="579bb08b-8f4a-4731-9a08-9a189e7c0778" containerName="revoke-keys" Sep 05 06:44:01.017300 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.017283 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.021501 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.021479 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-39047b-openshift-default-dockercfg-wfclr\"" Sep 05 06:44:01.035506 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.035482 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx"] Sep 05 06:44:01.068547 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.068514 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-workload-socket\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.068731 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.068570 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec947db4-e6a6-4f5e-af7d-58712a785965-istio-podinfo\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.068731 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.068596 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec947db4-e6a6-4f5e-af7d-58712a785965-istiod-ca-cert\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.068731 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.068629 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec947db4-e6a6-4f5e-af7d-58712a785965-istio-token\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.068945 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.068744 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-istio-envoy\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.068945 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.068782 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-credential-socket\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.068945 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.068855 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-workload-certs\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.068945 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.068898 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-istio-data\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.068945 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.068918 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26nn5\" (UniqueName: \"kubernetes.io/projected/ec947db4-e6a6-4f5e-af7d-58712a785965-kube-api-access-26nn5\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.169859 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.169814 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-istio-envoy\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.169859 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.169858 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-credential-socket\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.170117 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.169896 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-workload-certs\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.170117 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.169929 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-istio-data\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.170117 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.169955 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26nn5\" (UniqueName: \"kubernetes.io/projected/ec947db4-e6a6-4f5e-af7d-58712a785965-kube-api-access-26nn5\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.170117 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.170004 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-workload-socket\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.170117 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.170048 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec947db4-e6a6-4f5e-af7d-58712a785965-istio-podinfo\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.170117 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.170070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec947db4-e6a6-4f5e-af7d-58712a785965-istiod-ca-cert\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.170442 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.170123 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec947db4-e6a6-4f5e-af7d-58712a785965-istio-token\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.170442 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.170329 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-workload-certs\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.170442 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.170377 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-credential-socket\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.170614 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.170589 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-workload-socket\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.170614 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.170604 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-istio-data\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.170837 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.170821 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec947db4-e6a6-4f5e-af7d-58712a785965-istiod-ca-cert\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.172392 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.172368 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-istio-envoy\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.172747 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.172728 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec947db4-e6a6-4f5e-af7d-58712a785965-istio-podinfo\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.179166 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.179139 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec947db4-e6a6-4f5e-af7d-58712a785965-istio-token\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.179257 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.179245 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26nn5\" (UniqueName: \"kubernetes.io/projected/ec947db4-e6a6-4f5e-af7d-58712a785965-kube-api-access-26nn5\") pod \"e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.327641 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.327544 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:01.606473 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.606390 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx"] Sep 05 06:44:01.608949 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:44:01.608917 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec947db4_e6a6_4f5e_af7d_58712a785965.slice/crio-7c5f9f14c30c682ef61319e6cbd16982c709ad2402a5e7f299655cc34efbd0ac WatchSource:0}: Error finding container 7c5f9f14c30c682ef61319e6cbd16982c709ad2402a5e7f299655cc34efbd0ac: Status 404 returned error can't find the container with id 7c5f9f14c30c682ef61319e6cbd16982c709ad2402a5e7f299655cc34efbd0ac Sep 05 06:44:01.611702 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.611663 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:44:01.611795 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.611755 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:44:01.611844 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:01.611797 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:44:02.614132 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:02.614095 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" event={"ID":"ec947db4-e6a6-4f5e-af7d-58712a785965","Type":"ContainerStarted","Data":"3602b5519f99ad970f394f829fd75e8448e40bd5e66d4710c8f73cbbec1d3efb"} Sep 05 06:44:02.614592 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:02.614150 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" event={"ID":"ec947db4-e6a6-4f5e-af7d-58712a785965","Type":"ContainerStarted","Data":"7c5f9f14c30c682ef61319e6cbd16982c709ad2402a5e7f299655cc34efbd0ac"} Sep 05 06:44:02.637368 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:02.637297 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" podStartSLOduration=2.637279661 podStartE2EDuration="2.637279661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:44:02.636371419 +0000 UTC m=+1067.020914608" watchObservedRunningTime="2026-09-05 06:44:02.637279661 +0000 UTC m=+1067.021822785" Sep 05 06:44:03.327733 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:03.327697 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:03.332650 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:03.332622 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:03.620577 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:03.620548 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:03.621660 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:03.621642 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:05.237524 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:05.237489 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw"] Sep 05 06:44:05.237988 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:05.237706 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" podUID="99e28eb1-8011-4e71-a427-07c65bf41750" containerName="istio-proxy" containerID="cri-o://1ceb0b4dd9b8562d166e59e1eb57a4c0e1c3852798ffa2f48e9f816dfb306752" gracePeriod=30 Sep 05 06:44:06.762900 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:06.762808 2565 patch_prober.go:28] interesting pod/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 05 06:44:06.762900 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:06.762884 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" podUID="99e28eb1-8011-4e71-a427-07c65bf41750" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 05 06:44:10.507577 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.507517 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:44:10.571486 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.571459 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e28eb1-8011-4e71-a427-07c65bf41750-istiod-ca-cert\") pod \"99e28eb1-8011-4e71-a427-07c65bf41750\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " Sep 05 06:44:10.571486 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.571491 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-workload-certs\") pod \"99e28eb1-8011-4e71-a427-07c65bf41750\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " Sep 05 06:44:10.571714 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.571519 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-workload-socket\") pod \"99e28eb1-8011-4e71-a427-07c65bf41750\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " Sep 05 06:44:10.571714 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.571562 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e28eb1-8011-4e71-a427-07c65bf41750-istio-podinfo\") pod \"99e28eb1-8011-4e71-a427-07c65bf41750\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " Sep 05 06:44:10.571714 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.571601 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-credential-socket\") pod \"99e28eb1-8011-4e71-a427-07c65bf41750\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " Sep 05 06:44:10.571714 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.571628 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e28eb1-8011-4e71-a427-07c65bf41750-istio-token\") pod \"99e28eb1-8011-4e71-a427-07c65bf41750\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " Sep 05 06:44:10.571714 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.571653 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-istio-envoy\") pod \"99e28eb1-8011-4e71-a427-07c65bf41750\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " Sep 05 06:44:10.571714 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.571687 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhn9v\" (UniqueName: \"kubernetes.io/projected/99e28eb1-8011-4e71-a427-07c65bf41750-kube-api-access-zhn9v\") pod \"99e28eb1-8011-4e71-a427-07c65bf41750\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " Sep 05 06:44:10.571995 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.571747 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-istio-data\") pod \"99e28eb1-8011-4e71-a427-07c65bf41750\" (UID: \"99e28eb1-8011-4e71-a427-07c65bf41750\") " Sep 05 06:44:10.571995 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.571763 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "99e28eb1-8011-4e71-a427-07c65bf41750" (UID: "99e28eb1-8011-4e71-a427-07c65bf41750"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:44:10.571995 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.571861 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99e28eb1-8011-4e71-a427-07c65bf41750-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "99e28eb1-8011-4e71-a427-07c65bf41750" (UID: "99e28eb1-8011-4e71-a427-07c65bf41750"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:44:10.571995 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.571958 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "99e28eb1-8011-4e71-a427-07c65bf41750" (UID: "99e28eb1-8011-4e71-a427-07c65bf41750"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:44:10.572190 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.572071 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-istio-data" (OuterVolumeSpecName: "istio-data") pod "99e28eb1-8011-4e71-a427-07c65bf41750" (UID: "99e28eb1-8011-4e71-a427-07c65bf41750"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:44:10.572190 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.572123 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-credential-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:44:10.572190 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.572144 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e28eb1-8011-4e71-a427-07c65bf41750-istiod-ca-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:44:10.572190 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.572159 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-workload-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:44:10.572444 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.572248 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "99e28eb1-8011-4e71-a427-07c65bf41750" (UID: "99e28eb1-8011-4e71-a427-07c65bf41750"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:44:10.574050 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.574018 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "99e28eb1-8011-4e71-a427-07c65bf41750" (UID: "99e28eb1-8011-4e71-a427-07c65bf41750"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:44:10.574210 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.574186 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e28eb1-8011-4e71-a427-07c65bf41750-kube-api-access-zhn9v" (OuterVolumeSpecName: "kube-api-access-zhn9v") pod "99e28eb1-8011-4e71-a427-07c65bf41750" (UID: "99e28eb1-8011-4e71-a427-07c65bf41750"). InnerVolumeSpecName "kube-api-access-zhn9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:44:10.574417 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.574399 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99e28eb1-8011-4e71-a427-07c65bf41750-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "99e28eb1-8011-4e71-a427-07c65bf41750" (UID: "99e28eb1-8011-4e71-a427-07c65bf41750"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 05 06:44:10.574461 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.574419 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e28eb1-8011-4e71-a427-07c65bf41750-istio-token" (OuterVolumeSpecName: "istio-token") pod "99e28eb1-8011-4e71-a427-07c65bf41750" (UID: "99e28eb1-8011-4e71-a427-07c65bf41750"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:44:10.671578 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.671531 2565 generic.go:356] "Generic (PLEG): container finished" podID="99e28eb1-8011-4e71-a427-07c65bf41750" containerID="1ceb0b4dd9b8562d166e59e1eb57a4c0e1c3852798ffa2f48e9f816dfb306752" exitCode=0 Sep 05 06:44:10.671845 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.671581 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" event={"ID":"99e28eb1-8011-4e71-a427-07c65bf41750","Type":"ContainerDied","Data":"1ceb0b4dd9b8562d166e59e1eb57a4c0e1c3852798ffa2f48e9f816dfb306752"} Sep 05 06:44:10.671845 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.671619 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" Sep 05 06:44:10.671845 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.671652 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw" event={"ID":"99e28eb1-8011-4e71-a427-07c65bf41750","Type":"ContainerDied","Data":"1b55be209979703175402ec2a8f4e7cfca75cc99abb4b62dbb394fd7d6e3d9b3"} Sep 05 06:44:10.671845 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.671685 2565 scope.go:117] "RemoveContainer" containerID="1ceb0b4dd9b8562d166e59e1eb57a4c0e1c3852798ffa2f48e9f816dfb306752" Sep 05 06:44:10.672816 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.672790 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-workload-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:44:10.673728 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.672821 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e28eb1-8011-4e71-a427-07c65bf41750-istio-podinfo\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:44:10.673728 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.672834 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e28eb1-8011-4e71-a427-07c65bf41750-istio-token\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:44:10.673728 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.672845 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-istio-envoy\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:44:10.673728 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.672858 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhn9v\" (UniqueName: \"kubernetes.io/projected/99e28eb1-8011-4e71-a427-07c65bf41750-kube-api-access-zhn9v\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:44:10.673728 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.672872 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e28eb1-8011-4e71-a427-07c65bf41750-istio-data\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:44:10.686123 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.686087 2565 scope.go:117] "RemoveContainer" containerID="3b0644aa5f335fda3219f8efb3fc2d3bef066d24cd0f8810fe83938e0db83e91" Sep 05 06:44:10.695434 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.695414 2565 scope.go:117] "RemoveContainer" containerID="1ceb0b4dd9b8562d166e59e1eb57a4c0e1c3852798ffa2f48e9f816dfb306752" Sep 05 06:44:10.695747 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:44:10.695729 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ceb0b4dd9b8562d166e59e1eb57a4c0e1c3852798ffa2f48e9f816dfb306752\": container with ID starting with 1ceb0b4dd9b8562d166e59e1eb57a4c0e1c3852798ffa2f48e9f816dfb306752 not found: ID does not exist" containerID="1ceb0b4dd9b8562d166e59e1eb57a4c0e1c3852798ffa2f48e9f816dfb306752" Sep 05 06:44:10.695803 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.695759 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ceb0b4dd9b8562d166e59e1eb57a4c0e1c3852798ffa2f48e9f816dfb306752"} err="failed to get container status \"1ceb0b4dd9b8562d166e59e1eb57a4c0e1c3852798ffa2f48e9f816dfb306752\": rpc error: code = NotFound desc = could not find container \"1ceb0b4dd9b8562d166e59e1eb57a4c0e1c3852798ffa2f48e9f816dfb306752\": container with ID starting with 1ceb0b4dd9b8562d166e59e1eb57a4c0e1c3852798ffa2f48e9f816dfb306752 not found: ID does not exist" Sep 05 06:44:10.695803 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.695777 2565 scope.go:117] "RemoveContainer" containerID="3b0644aa5f335fda3219f8efb3fc2d3bef066d24cd0f8810fe83938e0db83e91" Sep 05 06:44:10.696083 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:44:10.696058 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b0644aa5f335fda3219f8efb3fc2d3bef066d24cd0f8810fe83938e0db83e91\": container with ID starting with 3b0644aa5f335fda3219f8efb3fc2d3bef066d24cd0f8810fe83938e0db83e91 not found: ID does not exist" containerID="3b0644aa5f335fda3219f8efb3fc2d3bef066d24cd0f8810fe83938e0db83e91" Sep 05 06:44:10.696193 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.696088 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0644aa5f335fda3219f8efb3fc2d3bef066d24cd0f8810fe83938e0db83e91"} err="failed to get container status \"3b0644aa5f335fda3219f8efb3fc2d3bef066d24cd0f8810fe83938e0db83e91\": rpc error: code = NotFound desc = could not find container \"3b0644aa5f335fda3219f8efb3fc2d3bef066d24cd0f8810fe83938e0db83e91\": container with ID starting with 3b0644aa5f335fda3219f8efb3fc2d3bef066d24cd0f8810fe83938e0db83e91 not found: ID does not exist" Sep 05 06:44:10.698981 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.698959 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw"] Sep 05 06:44:10.706191 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:10.706168 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-810a2bb6-openshift-default-f6d9659d4-hmttw"] Sep 05 06:44:12.165984 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:12.165951 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99e28eb1-8011-4e71-a427-07c65bf41750" path="/var/lib/kubelet/pods/99e28eb1-8011-4e71-a427-07c65bf41750/volumes" Sep 05 06:44:19.566979 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:19.566806 2565 patch_prober.go:28] interesting pod/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": read tcp 10.133.0.2:36688->10.133.0.56:15021: read: connection reset by peer" start-of-body= Sep 05 06:44:19.566979 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:19.566901 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" podUID="ec947db4-e6a6-4f5e-af7d-58712a785965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": read tcp 10.133.0.2:36688->10.133.0.56:15021: read: connection reset by peer" Sep 05 06:44:19.744272 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:19.744183 2565 generic.go:356] "Generic (PLEG): container finished" podID="ec947db4-e6a6-4f5e-af7d-58712a785965" containerID="3602b5519f99ad970f394f829fd75e8448e40bd5e66d4710c8f73cbbec1d3efb" exitCode=0 Sep 05 06:44:19.744272 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:19.744260 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" event={"ID":"ec947db4-e6a6-4f5e-af7d-58712a785965","Type":"ContainerDied","Data":"3602b5519f99ad970f394f829fd75e8448e40bd5e66d4710c8f73cbbec1d3efb"} Sep 05 06:44:19.744828 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:19.744814 2565 scope.go:117] "RemoveContainer" containerID="3602b5519f99ad970f394f829fd75e8448e40bd5e66d4710c8f73cbbec1d3efb" Sep 05 06:44:19.745991 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:19.745965 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:44:19.746060 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:19.746046 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:44:19.746096 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:19.746081 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:44:20.178487 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.178446 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh"] Sep 05 06:44:20.179065 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.179045 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99e28eb1-8011-4e71-a427-07c65bf41750" containerName="istio-proxy" Sep 05 06:44:20.179155 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.179070 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="99e28eb1-8011-4e71-a427-07c65bf41750" containerName="istio-proxy" Sep 05 06:44:20.179193 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.179187 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="99e28eb1-8011-4e71-a427-07c65bf41750" containerName="istio-proxy" Sep 05 06:44:20.181607 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.181579 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.183913 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.183886 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-362c80-openshift-default-dockercfg-9gw2t\"" Sep 05 06:44:20.193755 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.193733 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh"] Sep 05 06:44:20.279982 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.279931 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n64vl\" (UniqueName: \"kubernetes.io/projected/f5557927-c8c3-4c86-a170-fe5b41cc23e8-kube-api-access-n64vl\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.280192 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.279993 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istio-data\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.280192 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.280105 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istio-envoy\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.280365 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.280247 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-workload-certs\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.280433 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.280384 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istiod-ca-cert\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.280502 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.280480 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-credential-socket\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.280555 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.280530 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istio-token\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.280602 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.280580 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-workload-socket\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.280669 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.280632 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istio-podinfo\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.382287 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.382237 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n64vl\" (UniqueName: \"kubernetes.io/projected/f5557927-c8c3-4c86-a170-fe5b41cc23e8-kube-api-access-n64vl\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.382519 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.382298 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istio-data\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.382519 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.382367 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istio-envoy\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.382519 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.382425 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-workload-certs\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.382519 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.382471 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istiod-ca-cert\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.382519 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.382517 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-credential-socket\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.382796 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.382556 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istio-token\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.382796 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.382607 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-workload-socket\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.382796 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.382661 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istio-podinfo\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.383700 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.383666 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-workload-socket\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.384563 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.383921 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istio-data\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.384563 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.384023 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istiod-ca-cert\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.384563 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.384273 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-workload-certs\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.384563 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.384520 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-credential-socket\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.394346 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.390977 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istio-envoy\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.394346 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.391266 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istio-podinfo\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.394791 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.394762 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n64vl\" (UniqueName: \"kubernetes.io/projected/f5557927-c8c3-4c86-a170-fe5b41cc23e8-kube-api-access-n64vl\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.395550 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.395520 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istio-token\") pod \"e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.492882 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.492469 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:20.754677 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.754601 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" event={"ID":"ec947db4-e6a6-4f5e-af7d-58712a785965","Type":"ContainerStarted","Data":"71d5a2550c5ca2cd73b7eec2baae51144536d19a91e5b2dd646141c2a1af9d77"} Sep 05 06:44:20.775925 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.775886 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh"] Sep 05 06:44:20.783324 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.783266 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:44:20.783449 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.783397 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:44:20.783449 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:20.783441 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:44:21.327784 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:21.327751 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:21.328006 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:21.327947 2565 patch_prober.go:28] interesting pod/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 05 06:44:21.328089 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:21.327995 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" podUID="ec947db4-e6a6-4f5e-af7d-58712a785965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 05 06:44:21.761865 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:21.761822 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" event={"ID":"f5557927-c8c3-4c86-a170-fe5b41cc23e8","Type":"ContainerStarted","Data":"d0c44356d7cea0f848ce1e1c3b97959dcba6433beb78cf104206d8b9ce50cf38"} Sep 05 06:44:21.762280 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:21.761883 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" event={"ID":"f5557927-c8c3-4c86-a170-fe5b41cc23e8","Type":"ContainerStarted","Data":"6cf64688149875075871f9508056670a23512aea5c9b5495b0ce4a61cfcbbb34"} Sep 05 06:44:21.785693 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:21.785629 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" podStartSLOduration=1.785608169 podStartE2EDuration="1.785608169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:44:21.782399182 +0000 UTC m=+1086.166942306" watchObservedRunningTime="2026-09-05 06:44:21.785608169 +0000 UTC m=+1086.170151293" Sep 05 06:44:22.328959 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:22.328911 2565 patch_prober.go:28] interesting pod/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 05 06:44:22.329154 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:22.328979 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" podUID="ec947db4-e6a6-4f5e-af7d-58712a785965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 05 06:44:22.495367 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:22.495326 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:22.503072 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:22.503037 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:22.767584 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:22.767550 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:22.768637 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:22.768619 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:23.328923 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:23.328885 2565 patch_prober.go:28] interesting pod/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 05 06:44:23.329112 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:23.328945 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" podUID="ec947db4-e6a6-4f5e-af7d-58712a785965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 05 06:44:24.328089 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:24.328053 2565 patch_prober.go:28] interesting pod/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 05 06:44:24.328555 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:24.328111 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" podUID="ec947db4-e6a6-4f5e-af7d-58712a785965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 05 06:44:25.328743 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:25.328704 2565 patch_prober.go:28] interesting pod/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 05 06:44:25.329144 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:25.328766 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" podUID="ec947db4-e6a6-4f5e-af7d-58712a785965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 05 06:44:25.618255 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:25.618220 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph"] Sep 05 06:44:25.618648 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:25.618634 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99e28eb1-8011-4e71-a427-07c65bf41750" containerName="istio-proxy" Sep 05 06:44:25.618697 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:25.618650 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="99e28eb1-8011-4e71-a427-07c65bf41750" containerName="istio-proxy" Sep 05 06:44:25.618740 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:25.618726 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="99e28eb1-8011-4e71-a427-07c65bf41750" containerName="istio-proxy" Sep 05 06:44:25.620957 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:25.620941 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph" Sep 05 06:44:25.623156 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:25.623137 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-362c80\"" Sep 05 06:44:25.632241 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:25.632220 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph"] Sep 05 06:44:25.639148 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:25.639118 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29bb320e-e078-4212-9542-0640a6e2181e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph\" (UID: \"29bb320e-e078-4212-9542-0640a6e2181e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph" Sep 05 06:44:25.639268 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:25.639201 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4m8g\" (UniqueName: \"kubernetes.io/projected/29bb320e-e078-4212-9542-0640a6e2181e-kube-api-access-z4m8g\") pod \"payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph\" (UID: \"29bb320e-e078-4212-9542-0640a6e2181e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph" Sep 05 06:44:25.740567 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:25.740531 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29bb320e-e078-4212-9542-0640a6e2181e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph\" (UID: \"29bb320e-e078-4212-9542-0640a6e2181e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph" Sep 05 06:44:25.740778 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:25.740619 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4m8g\" (UniqueName: \"kubernetes.io/projected/29bb320e-e078-4212-9542-0640a6e2181e-kube-api-access-z4m8g\") pod \"payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph\" (UID: \"29bb320e-e078-4212-9542-0640a6e2181e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph" Sep 05 06:44:25.742207 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:25.741958 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29bb320e-e078-4212-9542-0640a6e2181e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph\" (UID: \"29bb320e-e078-4212-9542-0640a6e2181e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph" Sep 05 06:44:25.751450 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:25.751426 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4m8g\" (UniqueName: \"kubernetes.io/projected/29bb320e-e078-4212-9542-0640a6e2181e-kube-api-access-z4m8g\") pod \"payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph\" (UID: \"29bb320e-e078-4212-9542-0640a6e2181e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph" Sep 05 06:44:25.931982 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:25.931880 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph" Sep 05 06:44:26.213800 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:26.208870 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph"] Sep 05 06:44:26.333992 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:26.333961 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:26.334459 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:26.334353 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:26.335164 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:26.335143 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:44:26.792347 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:26.792284 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph" event={"ID":"29bb320e-e078-4212-9542-0640a6e2181e","Type":"ContainerStarted","Data":"0db0488f66313ccbe2179b52ddcef53a8049a7437fa358f0631ec033328312cc"} Sep 05 06:44:26.792570 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:26.792373 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph" event={"ID":"29bb320e-e078-4212-9542-0640a6e2181e","Type":"ContainerStarted","Data":"b5a5a32e6834aae8cbad3f45c2424fba4b50d602e0d34c0c563834a166df3bd4"} Sep 05 06:44:26.812190 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:26.812139 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph" podStartSLOduration=1.812120326 podStartE2EDuration="1.812120326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:44:26.810087925 +0000 UTC m=+1091.194631048" watchObservedRunningTime="2026-09-05 06:44:26.812120326 +0000 UTC m=+1091.196663485" Sep 05 06:44:27.799224 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:27.799191 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph" Sep 05 06:44:38.807184 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:38.807153 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph" Sep 05 06:44:38.809457 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:38.809429 2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 05 06:44:38.809595 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:38.809488 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" podUID="f5557927-c8c3-4c86-a170-fe5b41cc23e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 05 06:44:39.868696 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:39.868660 2565 generic.go:356] "Generic (PLEG): container finished" podID="f5557927-c8c3-4c86-a170-fe5b41cc23e8" containerID="d0c44356d7cea0f848ce1e1c3b97959dcba6433beb78cf104206d8b9ce50cf38" exitCode=0 Sep 05 06:44:39.869230 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:39.868739 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" event={"ID":"f5557927-c8c3-4c86-a170-fe5b41cc23e8","Type":"ContainerDied","Data":"d0c44356d7cea0f848ce1e1c3b97959dcba6433beb78cf104206d8b9ce50cf38"} Sep 05 06:44:39.869298 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:39.869268 2565 scope.go:117] "RemoveContainer" containerID="d0c44356d7cea0f848ce1e1c3b97959dcba6433beb78cf104206d8b9ce50cf38" Sep 05 06:44:39.870455 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:39.870431 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:44:39.870533 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:39.870499 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:44:39.870573 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:39.870546 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:44:40.876910 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:40.876876 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" event={"ID":"f5557927-c8c3-4c86-a170-fe5b41cc23e8","Type":"ContainerStarted","Data":"361786cc72f60d15cecc92c6569485e9496f71820399f55ddfd58a86f2b06042"} Sep 05 06:44:41.493446 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:41.493385 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:41.493674 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:41.493584 2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 05 06:44:41.493674 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:41.493646 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" podUID="f5557927-c8c3-4c86-a170-fe5b41cc23e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 05 06:44:42.493982 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:42.493944 2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 05 06:44:42.494616 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:42.494584 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" podUID="f5557927-c8c3-4c86-a170-fe5b41cc23e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 05 06:44:43.493570 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:43.493523 2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 05 06:44:43.493789 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:43.493593 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" podUID="f5557927-c8c3-4c86-a170-fe5b41cc23e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 05 06:44:44.494130 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:44.494084 2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 05 06:44:44.494665 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:44.494183 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" podUID="f5557927-c8c3-4c86-a170-fe5b41cc23e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 05 06:44:45.493603 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:45.493563 2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 05 06:44:45.493899 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:45.493868 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" podUID="f5557927-c8c3-4c86-a170-fe5b41cc23e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 05 06:44:46.497289 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:46.497258 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:46.497763 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:46.497640 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:46.498282 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:46.498267 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:44:55.233382 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.233349 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d"] Sep 05 06:44:55.236726 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.236709 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d" Sep 05 06:44:55.239901 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.239880 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-5e47e8-dockercfg-5gkx9\"" Sep 05 06:44:55.241501 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.241482 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-5e47e8\"" Sep 05 06:44:55.251557 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.251537 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d"] Sep 05 06:44:55.331786 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.331736 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh"] Sep 05 06:44:55.336019 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.335994 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh" Sep 05 06:44:55.350388 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.350360 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh"] Sep 05 06:44:55.352482 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.352459 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmsz4\" (UniqueName: \"kubernetes.io/projected/11dde2b8-22a2-4bcb-aa75-65d15139c39e-kube-api-access-qmsz4\") pod \"payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d\" (UID: \"11dde2b8-22a2-4bcb-aa75-65d15139c39e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d" Sep 05 06:44:55.352600 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.352525 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11dde2b8-22a2-4bcb-aa75-65d15139c39e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d\" (UID: \"11dde2b8-22a2-4bcb-aa75-65d15139c39e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d" Sep 05 06:44:55.454038 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.454001 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11dde2b8-22a2-4bcb-aa75-65d15139c39e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d\" (UID: \"11dde2b8-22a2-4bcb-aa75-65d15139c39e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d" Sep 05 06:44:55.454221 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.454049 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a24b1e8-87a7-4106-b294-e4d0f6b75a0f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh\" (UID: \"7a24b1e8-87a7-4106-b294-e4d0f6b75a0f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh" Sep 05 06:44:55.454221 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.454117 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmsz4\" (UniqueName: \"kubernetes.io/projected/11dde2b8-22a2-4bcb-aa75-65d15139c39e-kube-api-access-qmsz4\") pod \"payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d\" (UID: \"11dde2b8-22a2-4bcb-aa75-65d15139c39e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d" Sep 05 06:44:55.454221 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.454205 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjgc5\" (UniqueName: \"kubernetes.io/projected/7a24b1e8-87a7-4106-b294-e4d0f6b75a0f-kube-api-access-xjgc5\") pod \"payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh\" (UID: \"7a24b1e8-87a7-4106-b294-e4d0f6b75a0f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh" Sep 05 06:44:55.454785 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.454768 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11dde2b8-22a2-4bcb-aa75-65d15139c39e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d\" (UID: \"11dde2b8-22a2-4bcb-aa75-65d15139c39e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d" Sep 05 06:44:55.465626 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.465595 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmsz4\" (UniqueName: \"kubernetes.io/projected/11dde2b8-22a2-4bcb-aa75-65d15139c39e-kube-api-access-qmsz4\") pod \"payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d\" (UID: \"11dde2b8-22a2-4bcb-aa75-65d15139c39e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d" Sep 05 06:44:55.547420 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.547290 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d" Sep 05 06:44:55.554793 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.554766 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a24b1e8-87a7-4106-b294-e4d0f6b75a0f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh\" (UID: \"7a24b1e8-87a7-4106-b294-e4d0f6b75a0f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh" Sep 05 06:44:55.554930 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.554913 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjgc5\" (UniqueName: \"kubernetes.io/projected/7a24b1e8-87a7-4106-b294-e4d0f6b75a0f-kube-api-access-xjgc5\") pod \"payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh\" (UID: \"7a24b1e8-87a7-4106-b294-e4d0f6b75a0f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh" Sep 05 06:44:55.555515 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.555490 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a24b1e8-87a7-4106-b294-e4d0f6b75a0f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh\" (UID: \"7a24b1e8-87a7-4106-b294-e4d0f6b75a0f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh" Sep 05 06:44:55.565081 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.565056 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjgc5\" (UniqueName: \"kubernetes.io/projected/7a24b1e8-87a7-4106-b294-e4d0f6b75a0f-kube-api-access-xjgc5\") pod \"payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh\" (UID: \"7a24b1e8-87a7-4106-b294-e4d0f6b75a0f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh" Sep 05 06:44:55.647489 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.647459 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh" Sep 05 06:44:55.827630 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.827597 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d"] Sep 05 06:44:55.829244 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:44:55.829217 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11dde2b8_22a2_4bcb_aa75_65d15139c39e.slice/crio-e76c33b47ab670bebc63af971949195fb764390c7a1281dec6ad928fc28ab937 WatchSource:0}: Error finding container e76c33b47ab670bebc63af971949195fb764390c7a1281dec6ad928fc28ab937: Status 404 returned error can't find the container with id e76c33b47ab670bebc63af971949195fb764390c7a1281dec6ad928fc28ab937 Sep 05 06:44:55.936517 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.936494 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh"] Sep 05 06:44:55.939421 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:44:55.939386 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a24b1e8_87a7_4106_b294_e4d0f6b75a0f.slice/crio-744265594f425e59f60dbc83f3b293c31bfa6d3368b479eb0bd8f2b5160faf6d WatchSource:0}: Error finding container 744265594f425e59f60dbc83f3b293c31bfa6d3368b479eb0bd8f2b5160faf6d: Status 404 returned error can't find the container with id 744265594f425e59f60dbc83f3b293c31bfa6d3368b479eb0bd8f2b5160faf6d Sep 05 06:44:55.972372 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.972341 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh" event={"ID":"7a24b1e8-87a7-4106-b294-e4d0f6b75a0f","Type":"ContainerStarted","Data":"744265594f425e59f60dbc83f3b293c31bfa6d3368b479eb0bd8f2b5160faf6d"} Sep 05 06:44:55.973965 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.973940 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d" event={"ID":"11dde2b8-22a2-4bcb-aa75-65d15139c39e","Type":"ContainerStarted","Data":"e785aa9b25ea882250609a035005a3cd1c8d7f1512bbb13b38c8b01a1d06934f"} Sep 05 06:44:55.974117 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.973997 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d" event={"ID":"11dde2b8-22a2-4bcb-aa75-65d15139c39e","Type":"ContainerStarted","Data":"e76c33b47ab670bebc63af971949195fb764390c7a1281dec6ad928fc28ab937"} Sep 05 06:44:55.974117 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.974028 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d" Sep 05 06:44:55.998858 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:55.998809 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d" podStartSLOduration=0.998792147 podStartE2EDuration="998.792147ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:44:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:44:55.996199196 +0000 UTC m=+1120.380742346" watchObservedRunningTime="2026-09-05 06:44:55.998792147 +0000 UTC m=+1120.383335271" Sep 05 06:44:56.981530 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:56.981483 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh" event={"ID":"7a24b1e8-87a7-4106-b294-e4d0f6b75a0f","Type":"ContainerStarted","Data":"a4e85180fca7041c527543e45390fe66de49092c524c1f05e70383814376d336"} Sep 05 06:44:56.982009 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:56.981560 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh" Sep 05 06:44:57.006582 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:44:57.006516 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh" podStartSLOduration=2.006492327 podStartE2EDuration="2.006492327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:44:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:44:57.003350559 +0000 UTC m=+1121.387893684" watchObservedRunningTime="2026-09-05 06:44:57.006492327 +0000 UTC m=+1121.391035452" Sep 05 06:45:00.145173 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:00.145138 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-183ae6-29809845-zhmnw"] Sep 05 06:45:00.149057 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:00.149034 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-183ae6-29809845-zhmnw" Sep 05 06:45:00.151629 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:00.151592 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slfqj\"" Sep 05 06:45:00.153628 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:00.153608 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e47e8-29809845-zbccv"] Sep 05 06:45:00.157085 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:00.157023 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e47e8-29809845-zbccv" Sep 05 06:45:00.158818 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:00.158799 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-183ae6-29809845-zhmnw"] Sep 05 06:45:00.168132 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:00.161833 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-183ae6-29809845-zhmnw" Sep 05 06:45:00.173386 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:00.173360 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e47e8-29809845-zbccv"] Sep 05 06:45:00.194593 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:00.194352 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e47e8-29809845-zbccv" Sep 05 06:45:00.446506 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:00.446483 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-183ae6-29809845-zhmnw"] Sep 05 06:45:00.449123 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:45:00.449080 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8b6536_fb94_4ff8_b811_c5b25fffbb54.slice/crio-a45acee695cfb07c87b13a874eb590c847d6b05674374f4faac199d33ec26e5b WatchSource:0}: Error finding container a45acee695cfb07c87b13a874eb590c847d6b05674374f4faac199d33ec26e5b: Status 404 returned error can't find the container with id a45acee695cfb07c87b13a874eb590c847d6b05674374f4faac199d33ec26e5b Sep 05 06:45:00.482370 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:00.482348 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e47e8-29809845-zbccv"] Sep 05 06:45:00.484521 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:45:00.484493 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d0caf4_5c6f_4d2f_91da_d7cb023a50bd.slice/crio-e0c7ed0475e47eed828423d611e36b2cbeda15ca8f1c287c691500b834d91e46 WatchSource:0}: Error finding container e0c7ed0475e47eed828423d611e36b2cbeda15ca8f1c287c691500b834d91e46: Status 404 returned error can't find the container with id e0c7ed0475e47eed828423d611e36b2cbeda15ca8f1c287c691500b834d91e46 Sep 05 06:45:01.006882 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:01.006771 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e47e8-29809845-zbccv" event={"ID":"23d0caf4-5c6f-4d2f-91da-d7cb023a50bd","Type":"ContainerStarted","Data":"4acabfb74830c66958a8c434c1bfa4e4fa28546de104ddff8174cc533a333e40"} Sep 05 06:45:01.006882 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:01.006830 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e47e8-29809845-zbccv" event={"ID":"23d0caf4-5c6f-4d2f-91da-d7cb023a50bd","Type":"ContainerStarted","Data":"e0c7ed0475e47eed828423d611e36b2cbeda15ca8f1c287c691500b834d91e46"} Sep 05 06:45:01.008364 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:01.008337 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-183ae6-29809845-zhmnw" event={"ID":"7f8b6536-fb94-4ff8-b811-c5b25fffbb54","Type":"ContainerStarted","Data":"175c677ee1ed1427e6e502728d81f2ed176f91a12c2094abfb33ad355240d459"} Sep 05 06:45:01.008496 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:01.008379 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-183ae6-29809845-zhmnw" event={"ID":"7f8b6536-fb94-4ff8-b811-c5b25fffbb54","Type":"ContainerStarted","Data":"a45acee695cfb07c87b13a874eb590c847d6b05674374f4faac199d33ec26e5b"} Sep 05 06:45:01.025366 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:01.025294 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e47e8-29809845-zbccv" podStartSLOduration=1.025276245 podStartE2EDuration="1.025276245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:45:01.024592829 +0000 UTC m=+1125.409135943" watchObservedRunningTime="2026-09-05 06:45:01.025276245 +0000 UTC m=+1125.409819369" Sep 05 06:45:01.042761 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:01.042696 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-183ae6-29809845-zhmnw" podStartSLOduration=1.042677721 podStartE2EDuration="1.042677721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:45:01.041244654 +0000 UTC m=+1125.425787778" watchObservedRunningTime="2026-09-05 06:45:01.042677721 +0000 UTC m=+1125.427220872" Sep 05 06:45:02.016560 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:02.016525 2565 generic.go:356] "Generic (PLEG): container finished" podID="7f8b6536-fb94-4ff8-b811-c5b25fffbb54" containerID="175c677ee1ed1427e6e502728d81f2ed176f91a12c2094abfb33ad355240d459" exitCode=0 Sep 05 06:45:02.016962 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:02.016573 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-183ae6-29809845-zhmnw" event={"ID":"7f8b6536-fb94-4ff8-b811-c5b25fffbb54","Type":"ContainerDied","Data":"175c677ee1ed1427e6e502728d81f2ed176f91a12c2094abfb33ad355240d459"} Sep 05 06:45:02.018321 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:02.018282 2565 generic.go:356] "Generic (PLEG): container finished" podID="23d0caf4-5c6f-4d2f-91da-d7cb023a50bd" containerID="4acabfb74830c66958a8c434c1bfa4e4fa28546de104ddff8174cc533a333e40" exitCode=7 Sep 05 06:45:02.018471 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:02.018358 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e47e8-29809845-zbccv" event={"ID":"23d0caf4-5c6f-4d2f-91da-d7cb023a50bd","Type":"ContainerDied","Data":"4acabfb74830c66958a8c434c1bfa4e4fa28546de104ddff8174cc533a333e40"} Sep 05 06:45:02.018543 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:02.018532 2565 scope.go:117] "RemoveContainer" containerID="4acabfb74830c66958a8c434c1bfa4e4fa28546de104ddff8174cc533a333e40" Sep 05 06:45:03.025188 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:03.025094 2565 generic.go:356] "Generic (PLEG): container finished" podID="23d0caf4-5c6f-4d2f-91da-d7cb023a50bd" containerID="8a9a5340ff6a8f7dbd0160789692a7b101f6ab52b0e7e2a40816f5405917f245" exitCode=7 Sep 05 06:45:03.025662 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:03.025183 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e47e8-29809845-zbccv" event={"ID":"23d0caf4-5c6f-4d2f-91da-d7cb023a50bd","Type":"ContainerDied","Data":"8a9a5340ff6a8f7dbd0160789692a7b101f6ab52b0e7e2a40816f5405917f245"} Sep 05 06:45:03.025662 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:03.025253 2565 scope.go:117] "RemoveContainer" containerID="4acabfb74830c66958a8c434c1bfa4e4fa28546de104ddff8174cc533a333e40" Sep 05 06:45:03.025662 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:03.025365 2565 scope.go:117] "RemoveContainer" containerID="8a9a5340ff6a8f7dbd0160789692a7b101f6ab52b0e7e2a40816f5405917f245" Sep 05 06:45:03.026039 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:45:03.025656 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-5e47e8-29809845-zbccv_odh-ai-gateway-infra(23d0caf4-5c6f-4d2f-91da-d7cb023a50bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e47e8-29809845-zbccv" podUID="23d0caf4-5c6f-4d2f-91da-d7cb023a50bd" Sep 05 06:45:03.152588 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:03.152558 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-183ae6-29809845-zhmnw" Sep 05 06:45:04.032434 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:04.032406 2565 scope.go:117] "RemoveContainer" containerID="8a9a5340ff6a8f7dbd0160789692a7b101f6ab52b0e7e2a40816f5405917f245" Sep 05 06:45:04.032855 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:45:04.032682 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-5e47e8-29809845-zbccv_odh-ai-gateway-infra(23d0caf4-5c6f-4d2f-91da-d7cb023a50bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e47e8-29809845-zbccv" podUID="23d0caf4-5c6f-4d2f-91da-d7cb023a50bd" Sep 05 06:45:04.033548 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:04.033527 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-183ae6-29809845-zhmnw" Sep 05 06:45:04.033548 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:04.033542 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-183ae6-29809845-zhmnw" event={"ID":"7f8b6536-fb94-4ff8-b811-c5b25fffbb54","Type":"ContainerDied","Data":"a45acee695cfb07c87b13a874eb590c847d6b05674374f4faac199d33ec26e5b"} Sep 05 06:45:04.033669 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:04.033581 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a45acee695cfb07c87b13a874eb590c847d6b05674374f4faac199d33ec26e5b" Sep 05 06:45:06.984182 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:06.984151 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d" Sep 05 06:45:07.989778 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:07.989746 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh" Sep 05 06:45:16.219119 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:16.219084 2565 scope.go:117] "RemoveContainer" containerID="9f144cfd168ec544d3a097ad9c393d3717844f841c8e381a1db3710070e6a1c1" Sep 05 06:45:19.160215 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:19.160176 2565 scope.go:117] "RemoveContainer" containerID="8a9a5340ff6a8f7dbd0160789692a7b101f6ab52b0e7e2a40816f5405917f245" Sep 05 06:45:20.126690 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:20.126650 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e47e8-29809845-zbccv" event={"ID":"23d0caf4-5c6f-4d2f-91da-d7cb023a50bd","Type":"ContainerStarted","Data":"db477351ad12de7cab7fdbcc6bfb956818da80b11ccacd473823c502c1a96454"} Sep 05 06:45:21.135100 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:21.135064 2565 generic.go:356] "Generic (PLEG): container finished" podID="23d0caf4-5c6f-4d2f-91da-d7cb023a50bd" containerID="db477351ad12de7cab7fdbcc6bfb956818da80b11ccacd473823c502c1a96454" exitCode=0 Sep 05 06:45:21.135532 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:21.135137 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e47e8-29809845-zbccv" event={"ID":"23d0caf4-5c6f-4d2f-91da-d7cb023a50bd","Type":"ContainerDied","Data":"db477351ad12de7cab7fdbcc6bfb956818da80b11ccacd473823c502c1a96454"} Sep 05 06:45:21.135532 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:21.135194 2565 scope.go:117] "RemoveContainer" containerID="8a9a5340ff6a8f7dbd0160789692a7b101f6ab52b0e7e2a40816f5405917f245" Sep 05 06:45:21.161385 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:21.161353 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e47e8-29809845-zbccv"] Sep 05 06:45:22.279737 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:22.279708 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e47e8-29809845-zbccv" Sep 05 06:45:23.150165 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:23.150123 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e47e8-29809845-zbccv" Sep 05 06:45:23.150165 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:23.150157 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e47e8-29809845-zbccv" event={"ID":"23d0caf4-5c6f-4d2f-91da-d7cb023a50bd","Type":"ContainerDied","Data":"e0c7ed0475e47eed828423d611e36b2cbeda15ca8f1c287c691500b834d91e46"} Sep 05 06:45:23.150425 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:23.150218 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0c7ed0475e47eed828423d611e36b2cbeda15ca8f1c287c691500b834d91e46" Sep 05 06:45:23.186735 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:23.186701 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e47e8-29809845-zbccv"] Sep 05 06:45:23.191073 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:23.191048 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e47e8-29809845-zbccv"] Sep 05 06:45:24.166196 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:24.166159 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23d0caf4-5c6f-4d2f-91da-d7cb023a50bd" path="/var/lib/kubelet/pods/23d0caf4-5c6f-4d2f-91da-d7cb023a50bd/volumes" Sep 05 06:45:38.167117 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:38.167046 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62"] Sep 05 06:45:38.167630 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:38.167474 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23d0caf4-5c6f-4d2f-91da-d7cb023a50bd" containerName="cleanup" Sep 05 06:45:38.167630 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:38.167486 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="23d0caf4-5c6f-4d2f-91da-d7cb023a50bd" containerName="cleanup" Sep 05 06:45:38.167630 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:38.167503 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f8b6536-fb94-4ff8-b811-c5b25fffbb54" containerName="cleanup" Sep 05 06:45:38.167630 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:38.167509 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f8b6536-fb94-4ff8-b811-c5b25fffbb54" containerName="cleanup" Sep 05 06:45:38.167630 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:38.167529 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23d0caf4-5c6f-4d2f-91da-d7cb023a50bd" containerName="cleanup" Sep 05 06:45:38.167630 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:38.167534 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="23d0caf4-5c6f-4d2f-91da-d7cb023a50bd" containerName="cleanup" Sep 05 06:45:38.167630 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:38.167595 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="23d0caf4-5c6f-4d2f-91da-d7cb023a50bd" containerName="cleanup" Sep 05 06:45:38.167630 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:38.167604 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="23d0caf4-5c6f-4d2f-91da-d7cb023a50bd" containerName="cleanup" Sep 05 06:45:38.167630 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:38.167612 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f8b6536-fb94-4ff8-b811-c5b25fffbb54" containerName="cleanup" Sep 05 06:45:38.167630 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:38.167617 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="23d0caf4-5c6f-4d2f-91da-d7cb023a50bd" containerName="cleanup" Sep 05 06:45:38.171933 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:38.171914 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62" Sep 05 06:45:38.174477 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:38.174450 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-87058ae4-dockercfg-4qw8r\"" Sep 05 06:45:38.174798 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:38.174785 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-87058ae4\"" Sep 05 06:45:38.179985 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:38.179962 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62"] Sep 05 06:45:38.193627 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:38.193582 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46pnz\" (UniqueName: \"kubernetes.io/projected/3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49-kube-api-access-46pnz\") pod \"payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62\" (UID: \"3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49\") " pod="openshift-ingress/payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62" Sep 05 06:45:38.193800 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:38.193646 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49-plugins-config-volume\") pod \"payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62\" (UID: \"3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49\") " pod="openshift-ingress/payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62" Sep 05 06:45:38.295167 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:38.295125 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46pnz\" (UniqueName: \"kubernetes.io/projected/3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49-kube-api-access-46pnz\") pod \"payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62\" (UID: \"3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49\") " pod="openshift-ingress/payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62" Sep 05 06:45:38.295488 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:38.295465 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49-plugins-config-volume\") pod \"payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62\" (UID: \"3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49\") " pod="openshift-ingress/payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62" Sep 05 06:45:38.296349 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:38.296302 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49-plugins-config-volume\") pod \"payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62\" (UID: \"3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49\") " pod="openshift-ingress/payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62" Sep 05 06:45:38.306837 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:38.306809 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46pnz\" (UniqueName: \"kubernetes.io/projected/3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49-kube-api-access-46pnz\") pod \"payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62\" (UID: \"3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49\") " pod="openshift-ingress/payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62" Sep 05 06:45:38.483616 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:38.483525 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62" Sep 05 06:45:38.759335 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:38.759291 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62"] Sep 05 06:45:38.761493 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:45:38.761467 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af5ceb1_4b9c_41a1_aa43_1ef4fb8f8d49.slice/crio-c79441a5bfc613fb8b97c0aca744e2d30fa063e71d0b54615ab932cfe585c957 WatchSource:0}: Error finding container c79441a5bfc613fb8b97c0aca744e2d30fa063e71d0b54615ab932cfe585c957: Status 404 returned error can't find the container with id c79441a5bfc613fb8b97c0aca744e2d30fa063e71d0b54615ab932cfe585c957 Sep 05 06:45:39.247895 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:39.247855 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62" event={"ID":"3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49","Type":"ContainerStarted","Data":"75d562764c6b1b67524b0ba3a8567af86a4347fff0ee0c18e0221da50004f90a"} Sep 05 06:45:39.248297 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:39.247919 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62" event={"ID":"3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49","Type":"ContainerStarted","Data":"c79441a5bfc613fb8b97c0aca744e2d30fa063e71d0b54615ab932cfe585c957"} Sep 05 06:45:39.248297 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:39.247965 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62" Sep 05 06:45:39.266923 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:39.266862 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62" podStartSLOduration=1.266841935 podStartE2EDuration="1.266841935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:45:39.264585881 +0000 UTC m=+1163.649129002" watchObservedRunningTime="2026-09-05 06:45:39.266841935 +0000 UTC m=+1163.651385085" Sep 05 06:45:50.257072 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:45:50.257043 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62" Sep 05 06:47:01.702034 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:01.702003 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87058ae4-jbqht"] Sep 05 06:47:01.702600 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:01.702581 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23d0caf4-5c6f-4d2f-91da-d7cb023a50bd" containerName="cleanup" Sep 05 06:47:01.702654 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:01.702604 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="23d0caf4-5c6f-4d2f-91da-d7cb023a50bd" containerName="cleanup" Sep 05 06:47:01.706488 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:01.706469 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87058ae4-jbqht" Sep 05 06:47:01.709232 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:01.709213 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slfqj\"" Sep 05 06:47:01.709542 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:01.709523 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 05 06:47:01.714473 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:01.714443 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87058ae4-jbqht"] Sep 05 06:47:01.760716 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:01.760677 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5394318-4ac0-459b-8f3d-d53697549487-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-87058ae4-jbqht\" (UID: \"b5394318-4ac0-459b-8f3d-d53697549487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87058ae4-jbqht" Sep 05 06:47:01.862346 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:01.862271 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5394318-4ac0-459b-8f3d-d53697549487-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-87058ae4-jbqht\" (UID: \"b5394318-4ac0-459b-8f3d-d53697549487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87058ae4-jbqht" Sep 05 06:47:01.862931 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:01.862909 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5394318-4ac0-459b-8f3d-d53697549487-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-87058ae4-jbqht\" (UID: \"b5394318-4ac0-459b-8f3d-d53697549487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87058ae4-jbqht" Sep 05 06:47:02.020018 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:02.019932 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87058ae4-jbqht" Sep 05 06:47:02.296748 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:02.296659 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87058ae4-jbqht"] Sep 05 06:47:02.300076 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:47:02.300047 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5394318_4ac0_459b_8f3d_d53697549487.slice/crio-cfbf9128198fe57e3eeb0a14e846a563993dc71a466b8787b8dbdb8de73c6d5b WatchSource:0}: Error finding container cfbf9128198fe57e3eeb0a14e846a563993dc71a466b8787b8dbdb8de73c6d5b: Status 404 returned error can't find the container with id cfbf9128198fe57e3eeb0a14e846a563993dc71a466b8787b8dbdb8de73c6d5b Sep 05 06:47:02.717480 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:02.717439 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87058ae4-jbqht" event={"ID":"b5394318-4ac0-459b-8f3d-d53697549487","Type":"ContainerStarted","Data":"70fc5faee1a5a65aaff44135f43626ee5583e9fded36f7b2aa748f5c9b2432f9"} Sep 05 06:47:02.717912 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:02.717506 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87058ae4-jbqht" event={"ID":"b5394318-4ac0-459b-8f3d-d53697549487","Type":"ContainerStarted","Data":"cfbf9128198fe57e3eeb0a14e846a563993dc71a466b8787b8dbdb8de73c6d5b"} Sep 05 06:47:02.735685 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:02.735633 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87058ae4-jbqht" podStartSLOduration=1.735617177 podStartE2EDuration="1.735617177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:47:02.734076923 +0000 UTC m=+1247.118620046" watchObservedRunningTime="2026-09-05 06:47:02.735617177 +0000 UTC m=+1247.120160301" Sep 05 06:47:03.730578 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:03.730534 2565 generic.go:356] "Generic (PLEG): container finished" podID="b5394318-4ac0-459b-8f3d-d53697549487" containerID="70fc5faee1a5a65aaff44135f43626ee5583e9fded36f7b2aa748f5c9b2432f9" exitCode=0 Sep 05 06:47:03.731053 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:03.730589 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87058ae4-jbqht" event={"ID":"b5394318-4ac0-459b-8f3d-d53697549487","Type":"ContainerDied","Data":"70fc5faee1a5a65aaff44135f43626ee5583e9fded36f7b2aa748f5c9b2432f9"} Sep 05 06:47:04.879011 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:04.878987 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87058ae4-jbqht" Sep 05 06:47:04.997702 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:04.997612 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5394318-4ac0-459b-8f3d-d53697549487-maas-api-service-ca\") pod \"b5394318-4ac0-459b-8f3d-d53697549487\" (UID: \"b5394318-4ac0-459b-8f3d-d53697549487\") " Sep 05 06:47:04.997987 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:04.997962 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5394318-4ac0-459b-8f3d-d53697549487-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b5394318-4ac0-459b-8f3d-d53697549487" (UID: "b5394318-4ac0-459b-8f3d-d53697549487"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:47:05.098745 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:05.098709 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5394318-4ac0-459b-8f3d-d53697549487-maas-api-service-ca\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:47:05.744467 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:05.744436 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87058ae4-jbqht" Sep 05 06:47:05.744649 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:05.744473 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87058ae4-jbqht" event={"ID":"b5394318-4ac0-459b-8f3d-d53697549487","Type":"ContainerDied","Data":"cfbf9128198fe57e3eeb0a14e846a563993dc71a466b8787b8dbdb8de73c6d5b"} Sep 05 06:47:05.744649 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:05.744516 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfbf9128198fe57e3eeb0a14e846a563993dc71a466b8787b8dbdb8de73c6d5b" Sep 05 06:47:16.253716 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:16.253639 2565 scope.go:117] "RemoveContainer" containerID="54e73981fb191f9d4bac05af806bb7aa107805d48d0435d159f0678ff735f052" Sep 05 06:47:17.292610 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:17.292571 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62"] Sep 05 06:47:17.293046 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:17.292847 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62" podUID="3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49" containerName="payload-processing" containerID="cri-o://75d562764c6b1b67524b0ba3a8567af86a4347fff0ee0c18e0221da50004f90a" gracePeriod=30 Sep 05 06:47:17.498824 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:17.498763 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87058ae4-jbqht"] Sep 05 06:47:17.502016 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:17.501970 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87058ae4-jbqht"] Sep 05 06:47:17.578821 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:17.578782 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62" Sep 05 06:47:17.726349 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:17.726263 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49-plugins-config-volume\") pod \"3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49\" (UID: \"3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49\") " Sep 05 06:47:17.726564 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:17.726497 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46pnz\" (UniqueName: \"kubernetes.io/projected/3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49-kube-api-access-46pnz\") pod \"3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49\" (UID: \"3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49\") " Sep 05 06:47:17.726693 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:17.726668 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49" (UID: "3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:47:17.726851 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:17.726834 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49-plugins-config-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:47:17.728790 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:17.728760 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49-kube-api-access-46pnz" (OuterVolumeSpecName: "kube-api-access-46pnz") pod "3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49" (UID: "3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49"). InnerVolumeSpecName "kube-api-access-46pnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:47:17.812997 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:17.812914 2565 generic.go:356] "Generic (PLEG): container finished" podID="3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49" containerID="75d562764c6b1b67524b0ba3a8567af86a4347fff0ee0c18e0221da50004f90a" exitCode=0 Sep 05 06:47:17.812997 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:17.812956 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62" Sep 05 06:47:17.813176 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:17.812996 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62" event={"ID":"3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49","Type":"ContainerDied","Data":"75d562764c6b1b67524b0ba3a8567af86a4347fff0ee0c18e0221da50004f90a"} Sep 05 06:47:17.813176 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:17.813045 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62" event={"ID":"3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49","Type":"ContainerDied","Data":"c79441a5bfc613fb8b97c0aca744e2d30fa063e71d0b54615ab932cfe585c957"} Sep 05 06:47:17.813176 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:17.813066 2565 scope.go:117] "RemoveContainer" containerID="75d562764c6b1b67524b0ba3a8567af86a4347fff0ee0c18e0221da50004f90a" Sep 05 06:47:17.822793 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:17.822773 2565 scope.go:117] "RemoveContainer" containerID="75d562764c6b1b67524b0ba3a8567af86a4347fff0ee0c18e0221da50004f90a" Sep 05 06:47:17.823107 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:47:17.823087 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d562764c6b1b67524b0ba3a8567af86a4347fff0ee0c18e0221da50004f90a\": container with ID starting with 75d562764c6b1b67524b0ba3a8567af86a4347fff0ee0c18e0221da50004f90a not found: ID does not exist" containerID="75d562764c6b1b67524b0ba3a8567af86a4347fff0ee0c18e0221da50004f90a" Sep 05 06:47:17.823154 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:17.823117 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d562764c6b1b67524b0ba3a8567af86a4347fff0ee0c18e0221da50004f90a"} err="failed to get container status \"75d562764c6b1b67524b0ba3a8567af86a4347fff0ee0c18e0221da50004f90a\": rpc error: code = NotFound desc = could not find container \"75d562764c6b1b67524b0ba3a8567af86a4347fff0ee0c18e0221da50004f90a\": container with ID starting with 75d562764c6b1b67524b0ba3a8567af86a4347fff0ee0c18e0221da50004f90a not found: ID does not exist" Sep 05 06:47:17.827629 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:17.827609 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46pnz\" (UniqueName: \"kubernetes.io/projected/3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49-kube-api-access-46pnz\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:47:17.835163 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:17.835135 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62"] Sep 05 06:47:17.839794 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:17.839772 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-87058ae4-6967b45bb9-csr62"] Sep 05 06:47:18.167326 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:18.167269 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49" path="/var/lib/kubelet/pods/3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49/volumes" Sep 05 06:47:18.167735 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:47:18.167715 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5394318-4ac0-459b-8f3d-d53697549487" path="/var/lib/kubelet/pods/b5394318-4ac0-459b-8f3d-d53697549487/volumes" Sep 05 06:48:16.274978 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:48:16.274947 2565 scope.go:117] "RemoveContainer" containerID="6eec886018eaa4662575e8f0a04c62e3a797890462e2e62d5281ff1b239c0d9a" Sep 05 06:48:16.283588 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:48:16.283569 2565 scope.go:117] "RemoveContainer" containerID="3b45e7fbc25ed6fa33083958bfc33bf5d680f253e55238289a889b84a2e14777" Sep 05 06:49:16.313944 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:16.313918 2565 scope.go:117] "RemoveContainer" containerID="744c9af622132a7f3ebe6d751356d5bfd13feb2707b251d091020144140dc079" Sep 05 06:49:16.322577 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:16.322561 2565 scope.go:117] "RemoveContainer" containerID="afdc8457567f6cd2bf897248b6b40c97593db75bc2c138a713839e06ead0b916" Sep 05 06:49:25.470976 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:25.470941 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-96d352-574469d885-6tz2f"] Sep 05 06:49:25.471460 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:25.471365 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5394318-4ac0-459b-8f3d-d53697549487" containerName="revoke-keys" Sep 05 06:49:25.471460 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:25.471376 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5394318-4ac0-459b-8f3d-d53697549487" containerName="revoke-keys" Sep 05 06:49:25.471460 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:25.471401 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49" containerName="payload-processing" Sep 05 06:49:25.471460 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:25.471406 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49" containerName="payload-processing" Sep 05 06:49:25.471591 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:25.471470 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3af5ceb1-4b9c-41a1-aa43-1ef4fb8f8d49" containerName="payload-processing" Sep 05 06:49:25.471591 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:25.471480 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5394318-4ac0-459b-8f3d-d53697549487" containerName="revoke-keys" Sep 05 06:49:25.475905 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:25.475890 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-96d352-574469d885-6tz2f" Sep 05 06:49:25.478710 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:25.478687 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-96d352-dockercfg-xb85d\"" Sep 05 06:49:25.478967 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:25.478950 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-96d352\"" Sep 05 06:49:25.485205 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:25.485181 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-96d352-574469d885-6tz2f"] Sep 05 06:49:25.486620 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:25.486601 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcc03824-79df-441e-b185-ba9fa32dac38-plugins-config-volume\") pod \"payload-processing-e2e-api-a-96d352-574469d885-6tz2f\" (UID: \"bcc03824-79df-441e-b185-ba9fa32dac38\") " pod="openshift-ingress/payload-processing-e2e-api-a-96d352-574469d885-6tz2f" Sep 05 06:49:25.486701 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:25.486641 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wsdx\" (UniqueName: \"kubernetes.io/projected/bcc03824-79df-441e-b185-ba9fa32dac38-kube-api-access-6wsdx\") pod \"payload-processing-e2e-api-a-96d352-574469d885-6tz2f\" (UID: \"bcc03824-79df-441e-b185-ba9fa32dac38\") " pod="openshift-ingress/payload-processing-e2e-api-a-96d352-574469d885-6tz2f" Sep 05 06:49:25.587898 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:25.587846 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wsdx\" (UniqueName: \"kubernetes.io/projected/bcc03824-79df-441e-b185-ba9fa32dac38-kube-api-access-6wsdx\") pod \"payload-processing-e2e-api-a-96d352-574469d885-6tz2f\" (UID: \"bcc03824-79df-441e-b185-ba9fa32dac38\") " pod="openshift-ingress/payload-processing-e2e-api-a-96d352-574469d885-6tz2f" Sep 05 06:49:25.588089 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:25.588047 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcc03824-79df-441e-b185-ba9fa32dac38-plugins-config-volume\") pod \"payload-processing-e2e-api-a-96d352-574469d885-6tz2f\" (UID: \"bcc03824-79df-441e-b185-ba9fa32dac38\") " pod="openshift-ingress/payload-processing-e2e-api-a-96d352-574469d885-6tz2f" Sep 05 06:49:25.589113 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:25.589088 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcc03824-79df-441e-b185-ba9fa32dac38-plugins-config-volume\") pod \"payload-processing-e2e-api-a-96d352-574469d885-6tz2f\" (UID: \"bcc03824-79df-441e-b185-ba9fa32dac38\") " pod="openshift-ingress/payload-processing-e2e-api-a-96d352-574469d885-6tz2f" Sep 05 06:49:25.604723 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:25.604565 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wsdx\" (UniqueName: \"kubernetes.io/projected/bcc03824-79df-441e-b185-ba9fa32dac38-kube-api-access-6wsdx\") pod \"payload-processing-e2e-api-a-96d352-574469d885-6tz2f\" (UID: \"bcc03824-79df-441e-b185-ba9fa32dac38\") " pod="openshift-ingress/payload-processing-e2e-api-a-96d352-574469d885-6tz2f" Sep 05 06:49:25.787599 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:25.787508 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-96d352-574469d885-6tz2f" Sep 05 06:49:26.063425 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:26.063399 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-96d352-574469d885-6tz2f"] Sep 05 06:49:26.065582 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:49:26.065552 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc03824_79df_441e_b185_ba9fa32dac38.slice/crio-b0b2a59fa43036a0e224efdcca3e08d4c57233e7abd23a12b58a6dff2810d793 WatchSource:0}: Error finding container b0b2a59fa43036a0e224efdcca3e08d4c57233e7abd23a12b58a6dff2810d793: Status 404 returned error can't find the container with id b0b2a59fa43036a0e224efdcca3e08d4c57233e7abd23a12b58a6dff2810d793 Sep 05 06:49:26.522292 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:26.522251 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-96d352-574469d885-6tz2f" event={"ID":"bcc03824-79df-441e-b185-ba9fa32dac38","Type":"ContainerStarted","Data":"830c22fbb892a3854634996bdad1a6b4de00e56c7be472ba67de87c8535dc0df"} Sep 05 06:49:26.522701 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:26.522346 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-96d352-574469d885-6tz2f" event={"ID":"bcc03824-79df-441e-b185-ba9fa32dac38","Type":"ContainerStarted","Data":"b0b2a59fa43036a0e224efdcca3e08d4c57233e7abd23a12b58a6dff2810d793"} Sep 05 06:49:26.522701 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:26.522377 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-96d352-574469d885-6tz2f" Sep 05 06:49:26.543128 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:26.543086 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-96d352-574469d885-6tz2f" podStartSLOduration=1.5430679889999999 podStartE2EDuration="1.543067989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:49:26.541516472 +0000 UTC m=+1390.926059596" watchObservedRunningTime="2026-09-05 06:49:26.543067989 +0000 UTC m=+1390.927611112" Sep 05 06:49:37.529983 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:37.529950 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-96d352-574469d885-6tz2f" Sep 05 06:49:54.559428 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:54.559400 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8"] Sep 05 06:49:54.562967 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:54.562949 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8" Sep 05 06:49:54.565595 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:54.565575 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a9301f-dockercfg-q5pvn\"" Sep 05 06:49:54.565776 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:54.565763 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a9301f\"" Sep 05 06:49:54.574804 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:54.574754 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8"] Sep 05 06:49:54.576964 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:54.576943 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd"] Sep 05 06:49:54.580402 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:54.580386 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd" Sep 05 06:49:54.590368 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:54.590350 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd"] Sep 05 06:49:54.664398 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:54.664369 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/323ad275-3db9-43cf-96ea-ac33f6999051-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8\" (UID: \"323ad275-3db9-43cf-96ea-ac33f6999051\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8" Sep 05 06:49:54.664580 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:54.664410 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mqtx\" (UniqueName: \"kubernetes.io/projected/9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05-kube-api-access-7mqtx\") pod \"payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd\" (UID: \"9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd" Sep 05 06:49:54.664580 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:54.664458 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q56l4\" (UniqueName: \"kubernetes.io/projected/323ad275-3db9-43cf-96ea-ac33f6999051-kube-api-access-q56l4\") pod \"payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8\" (UID: \"323ad275-3db9-43cf-96ea-ac33f6999051\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8" Sep 05 06:49:54.664580 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:54.664485 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd\" (UID: \"9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd" Sep 05 06:49:54.765471 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:54.765431 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/323ad275-3db9-43cf-96ea-ac33f6999051-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8\" (UID: \"323ad275-3db9-43cf-96ea-ac33f6999051\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8" Sep 05 06:49:54.765471 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:54.765477 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mqtx\" (UniqueName: \"kubernetes.io/projected/9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05-kube-api-access-7mqtx\") pod \"payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd\" (UID: \"9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd" Sep 05 06:49:54.765738 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:54.765525 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q56l4\" (UniqueName: \"kubernetes.io/projected/323ad275-3db9-43cf-96ea-ac33f6999051-kube-api-access-q56l4\") pod \"payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8\" (UID: \"323ad275-3db9-43cf-96ea-ac33f6999051\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8" Sep 05 06:49:54.765738 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:54.765545 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd\" (UID: \"9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd" Sep 05 06:49:54.766195 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:54.766170 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/323ad275-3db9-43cf-96ea-ac33f6999051-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8\" (UID: \"323ad275-3db9-43cf-96ea-ac33f6999051\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8" Sep 05 06:49:54.766283 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:54.766236 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd\" (UID: \"9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd" Sep 05 06:49:54.774770 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:54.774748 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mqtx\" (UniqueName: \"kubernetes.io/projected/9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05-kube-api-access-7mqtx\") pod \"payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd\" (UID: \"9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd" Sep 05 06:49:54.774876 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:54.774861 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q56l4\" (UniqueName: \"kubernetes.io/projected/323ad275-3db9-43cf-96ea-ac33f6999051-kube-api-access-q56l4\") pod \"payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8\" (UID: \"323ad275-3db9-43cf-96ea-ac33f6999051\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8" Sep 05 06:49:54.874551 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:54.874525 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8" Sep 05 06:49:54.891968 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:54.891939 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd" Sep 05 06:49:55.150834 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:55.150744 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8"] Sep 05 06:49:55.154043 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:49:55.154015 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323ad275_3db9_43cf_96ea_ac33f6999051.slice/crio-dba3d5ab7f912fcd3c8757a8abaf087e16b3e154688b2cd7d4ae458390819dd4 WatchSource:0}: Error finding container dba3d5ab7f912fcd3c8757a8abaf087e16b3e154688b2cd7d4ae458390819dd4: Status 404 returned error can't find the container with id dba3d5ab7f912fcd3c8757a8abaf087e16b3e154688b2cd7d4ae458390819dd4 Sep 05 06:49:55.175659 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:55.175635 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd"] Sep 05 06:49:55.178165 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:49:55.178139 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f93a9a8_9bb5_4d12_ad80_f4d5dd8e8c05.slice/crio-12dae54960b2ab2c17ac9590b17796431a42a9f74350edb6be2f6e09545d6dfd WatchSource:0}: Error finding container 12dae54960b2ab2c17ac9590b17796431a42a9f74350edb6be2f6e09545d6dfd: Status 404 returned error can't find the container with id 12dae54960b2ab2c17ac9590b17796431a42a9f74350edb6be2f6e09545d6dfd Sep 05 06:49:55.686687 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:55.686625 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8" event={"ID":"323ad275-3db9-43cf-96ea-ac33f6999051","Type":"ContainerStarted","Data":"7938e76ffccd22d6d622ad1fcaa32e2e19750928940bafca178a5118abe1c657"} Sep 05 06:49:55.686687 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:55.686692 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8" Sep 05 06:49:55.687237 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:55.686715 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8" event={"ID":"323ad275-3db9-43cf-96ea-ac33f6999051","Type":"ContainerStarted","Data":"dba3d5ab7f912fcd3c8757a8abaf087e16b3e154688b2cd7d4ae458390819dd4"} Sep 05 06:49:55.688385 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:55.688360 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd" event={"ID":"9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05","Type":"ContainerStarted","Data":"8737db05d6e2b08a9bd44c9e4a32e167d229e30ee15d1949b2bf24f4e27625d4"} Sep 05 06:49:55.688504 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:55.688402 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd" event={"ID":"9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05","Type":"ContainerStarted","Data":"12dae54960b2ab2c17ac9590b17796431a42a9f74350edb6be2f6e09545d6dfd"} Sep 05 06:49:55.688504 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:55.688430 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd" Sep 05 06:49:55.704090 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:55.704045 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8" podStartSLOduration=1.704030505 podStartE2EDuration="1.704030505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:49:55.702013725 +0000 UTC m=+1420.086556850" watchObservedRunningTime="2026-09-05 06:49:55.704030505 +0000 UTC m=+1420.088573627" Sep 05 06:49:55.719477 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:49:55.719426 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd" podStartSLOduration=1.7194076790000001 podStartE2EDuration="1.719407679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:49:55.716646416 +0000 UTC m=+1420.101189554" watchObservedRunningTime="2026-09-05 06:49:55.719407679 +0000 UTC m=+1420.103950803" Sep 05 06:50:06.696146 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:06.696070 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd" Sep 05 06:50:06.696680 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:06.696155 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8" Sep 05 06:50:16.345179 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:16.345150 2565 scope.go:117] "RemoveContainer" containerID="b316fb9b7abb74fa5b4d94c2619e279de0a8ba4593168f3da12c51479683cbef" Sep 05 06:50:36.458936 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:36.458901 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-96d352-574469d885-6tz2f"] Sep 05 06:50:36.459437 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:36.459116 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-96d352-574469d885-6tz2f" podUID="bcc03824-79df-441e-b185-ba9fa32dac38" containerName="payload-processing" containerID="cri-o://830c22fbb892a3854634996bdad1a6b4de00e56c7be472ba67de87c8535dc0df" gracePeriod=30 Sep 05 06:50:36.727686 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:36.727609 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-96d352-574469d885-6tz2f" Sep 05 06:50:36.877574 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:36.877533 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcc03824-79df-441e-b185-ba9fa32dac38-plugins-config-volume\") pod \"bcc03824-79df-441e-b185-ba9fa32dac38\" (UID: \"bcc03824-79df-441e-b185-ba9fa32dac38\") " Sep 05 06:50:36.877783 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:36.877598 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wsdx\" (UniqueName: \"kubernetes.io/projected/bcc03824-79df-441e-b185-ba9fa32dac38-kube-api-access-6wsdx\") pod \"bcc03824-79df-441e-b185-ba9fa32dac38\" (UID: \"bcc03824-79df-441e-b185-ba9fa32dac38\") " Sep 05 06:50:36.877949 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:36.877926 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcc03824-79df-441e-b185-ba9fa32dac38-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bcc03824-79df-441e-b185-ba9fa32dac38" (UID: "bcc03824-79df-441e-b185-ba9fa32dac38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:50:36.879859 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:36.879829 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc03824-79df-441e-b185-ba9fa32dac38-kube-api-access-6wsdx" (OuterVolumeSpecName: "kube-api-access-6wsdx") pod "bcc03824-79df-441e-b185-ba9fa32dac38" (UID: "bcc03824-79df-441e-b185-ba9fa32dac38"). InnerVolumeSpecName "kube-api-access-6wsdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:50:36.916659 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:36.916632 2565 generic.go:356] "Generic (PLEG): container finished" podID="bcc03824-79df-441e-b185-ba9fa32dac38" containerID="830c22fbb892a3854634996bdad1a6b4de00e56c7be472ba67de87c8535dc0df" exitCode=0 Sep 05 06:50:36.916794 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:36.916704 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-96d352-574469d885-6tz2f" event={"ID":"bcc03824-79df-441e-b185-ba9fa32dac38","Type":"ContainerDied","Data":"830c22fbb892a3854634996bdad1a6b4de00e56c7be472ba67de87c8535dc0df"} Sep 05 06:50:36.916794 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:36.916725 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-96d352-574469d885-6tz2f" Sep 05 06:50:36.916794 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:36.916781 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-96d352-574469d885-6tz2f" event={"ID":"bcc03824-79df-441e-b185-ba9fa32dac38","Type":"ContainerDied","Data":"b0b2a59fa43036a0e224efdcca3e08d4c57233e7abd23a12b58a6dff2810d793"} Sep 05 06:50:36.916914 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:36.916810 2565 scope.go:117] "RemoveContainer" containerID="830c22fbb892a3854634996bdad1a6b4de00e56c7be472ba67de87c8535dc0df" Sep 05 06:50:36.925936 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:36.925921 2565 scope.go:117] "RemoveContainer" containerID="830c22fbb892a3854634996bdad1a6b4de00e56c7be472ba67de87c8535dc0df" Sep 05 06:50:36.926197 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:50:36.926179 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"830c22fbb892a3854634996bdad1a6b4de00e56c7be472ba67de87c8535dc0df\": container with ID starting with 830c22fbb892a3854634996bdad1a6b4de00e56c7be472ba67de87c8535dc0df not found: ID does not exist" containerID="830c22fbb892a3854634996bdad1a6b4de00e56c7be472ba67de87c8535dc0df" Sep 05 06:50:36.926246 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:36.926205 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"830c22fbb892a3854634996bdad1a6b4de00e56c7be472ba67de87c8535dc0df"} err="failed to get container status \"830c22fbb892a3854634996bdad1a6b4de00e56c7be472ba67de87c8535dc0df\": rpc error: code = NotFound desc = could not find container \"830c22fbb892a3854634996bdad1a6b4de00e56c7be472ba67de87c8535dc0df\": container with ID starting with 830c22fbb892a3854634996bdad1a6b4de00e56c7be472ba67de87c8535dc0df not found: ID does not exist" Sep 05 06:50:36.940086 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:36.940064 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-96d352-574469d885-6tz2f"] Sep 05 06:50:36.943952 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:36.943930 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-96d352-574469d885-6tz2f"] Sep 05 06:50:36.978547 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:36.978479 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcc03824-79df-441e-b185-ba9fa32dac38-plugins-config-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:50:36.978547 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:36.978500 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wsdx\" (UniqueName: \"kubernetes.io/projected/bcc03824-79df-441e-b185-ba9fa32dac38-kube-api-access-6wsdx\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:50:38.166474 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:38.166442 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcc03824-79df-441e-b185-ba9fa32dac38" path="/var/lib/kubelet/pods/bcc03824-79df-441e-b185-ba9fa32dac38/volumes" Sep 05 06:50:44.266154 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:44.266119 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9301f-6zl84"] Sep 05 06:50:44.266763 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:44.266741 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcc03824-79df-441e-b185-ba9fa32dac38" containerName="payload-processing" Sep 05 06:50:44.266850 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:44.266766 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcc03824-79df-441e-b185-ba9fa32dac38" containerName="payload-processing" Sep 05 06:50:44.266906 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:44.266899 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcc03824-79df-441e-b185-ba9fa32dac38" containerName="payload-processing" Sep 05 06:50:44.270041 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:44.270023 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9301f-6zl84" Sep 05 06:50:44.272367 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:44.272345 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slfqj\"" Sep 05 06:50:44.272468 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:44.272348 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 05 06:50:44.276782 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:44.276762 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9301f-6zl84"] Sep 05 06:50:44.449906 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:44.449865 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a1a7c4-6921-4ee8-843a-bcb71783db2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a9301f-6zl84\" (UID: \"b9a1a7c4-6921-4ee8-843a-bcb71783db2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9301f-6zl84" Sep 05 06:50:44.551106 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:44.551009 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a1a7c4-6921-4ee8-843a-bcb71783db2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a9301f-6zl84\" (UID: \"b9a1a7c4-6921-4ee8-843a-bcb71783db2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9301f-6zl84" Sep 05 06:50:44.551676 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:44.551657 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a1a7c4-6921-4ee8-843a-bcb71783db2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a9301f-6zl84\" (UID: \"b9a1a7c4-6921-4ee8-843a-bcb71783db2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9301f-6zl84" Sep 05 06:50:44.581912 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:44.581874 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9301f-6zl84" Sep 05 06:50:44.855452 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:44.855426 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9301f-6zl84"] Sep 05 06:50:44.857517 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:50:44.857485 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a1a7c4_6921_4ee8_843a_bcb71783db2e.slice/crio-3b485e22ba58a3a63d7ed3cb24750d9a742cfbeba433c82396d8d9406ba0c2c0 WatchSource:0}: Error finding container 3b485e22ba58a3a63d7ed3cb24750d9a742cfbeba433c82396d8d9406ba0c2c0: Status 404 returned error can't find the container with id 3b485e22ba58a3a63d7ed3cb24750d9a742cfbeba433c82396d8d9406ba0c2c0 Sep 05 06:50:44.970587 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:44.970551 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9301f-6zl84" event={"ID":"b9a1a7c4-6921-4ee8-843a-bcb71783db2e","Type":"ContainerStarted","Data":"3b485e22ba58a3a63d7ed3cb24750d9a742cfbeba433c82396d8d9406ba0c2c0"} Sep 05 06:50:45.977609 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:45.977573 2565 generic.go:356] "Generic (PLEG): container finished" podID="b9a1a7c4-6921-4ee8-843a-bcb71783db2e" containerID="bda61ad7dfe951c480dbd85fe1488b9a22c23d951bb74de9ded4440af69c9e5a" exitCode=0 Sep 05 06:50:45.977988 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:45.977659 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9301f-6zl84" event={"ID":"b9a1a7c4-6921-4ee8-843a-bcb71783db2e","Type":"ContainerDied","Data":"bda61ad7dfe951c480dbd85fe1488b9a22c23d951bb74de9ded4440af69c9e5a"} Sep 05 06:50:47.120579 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:47.120551 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9301f-6zl84" Sep 05 06:50:47.172803 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:47.172775 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a1a7c4-6921-4ee8-843a-bcb71783db2e-maas-api-service-ca\") pod \"b9a1a7c4-6921-4ee8-843a-bcb71783db2e\" (UID: \"b9a1a7c4-6921-4ee8-843a-bcb71783db2e\") " Sep 05 06:50:47.173091 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:47.173068 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a1a7c4-6921-4ee8-843a-bcb71783db2e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b9a1a7c4-6921-4ee8-843a-bcb71783db2e" (UID: "b9a1a7c4-6921-4ee8-843a-bcb71783db2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:50:47.274135 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:47.274048 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a1a7c4-6921-4ee8-843a-bcb71783db2e-maas-api-service-ca\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:50:47.990258 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:47.990224 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9301f-6zl84" Sep 05 06:50:47.990463 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:47.990255 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9301f-6zl84" event={"ID":"b9a1a7c4-6921-4ee8-843a-bcb71783db2e","Type":"ContainerDied","Data":"3b485e22ba58a3a63d7ed3cb24750d9a742cfbeba433c82396d8d9406ba0c2c0"} Sep 05 06:50:47.990463 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:50:47.990352 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b485e22ba58a3a63d7ed3cb24750d9a742cfbeba433c82396d8d9406ba0c2c0" Sep 05 06:51:00.210231 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:00.210195 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd"] Sep 05 06:51:00.210705 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:00.210477 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd" podUID="9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05" containerName="payload-processing" containerID="cri-o://8737db05d6e2b08a9bd44c9e4a32e167d229e30ee15d1949b2bf24f4e27625d4" gracePeriod=30 Sep 05 06:51:00.216816 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:00.216604 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8"] Sep 05 06:51:00.216947 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:00.216823 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8" podUID="323ad275-3db9-43cf-96ea-ac33f6999051" containerName="payload-pre-processing" containerID="cri-o://7938e76ffccd22d6d622ad1fcaa32e2e19750928940bafca178a5118abe1c657" gracePeriod=30 Sep 05 06:51:00.390548 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:00.390270 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9301f-6zl84"] Sep 05 06:51:00.393539 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:00.393510 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9301f-6zl84"] Sep 05 06:51:00.477859 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:00.477834 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd" Sep 05 06:51:00.510005 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:00.509981 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8" Sep 05 06:51:00.594610 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:00.594568 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mqtx\" (UniqueName: \"kubernetes.io/projected/9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05-kube-api-access-7mqtx\") pod \"9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05\" (UID: \"9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05\") " Sep 05 06:51:00.594809 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:00.594626 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05-plugins-config-volume\") pod \"9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05\" (UID: \"9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05\") " Sep 05 06:51:00.594809 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:00.594704 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/323ad275-3db9-43cf-96ea-ac33f6999051-plugins-config-volume\") pod \"323ad275-3db9-43cf-96ea-ac33f6999051\" (UID: \"323ad275-3db9-43cf-96ea-ac33f6999051\") " Sep 05 06:51:00.594809 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:00.594741 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q56l4\" (UniqueName: \"kubernetes.io/projected/323ad275-3db9-43cf-96ea-ac33f6999051-kube-api-access-q56l4\") pod \"323ad275-3db9-43cf-96ea-ac33f6999051\" (UID: \"323ad275-3db9-43cf-96ea-ac33f6999051\") " Sep 05 06:51:00.594987 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:00.594962 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05" (UID: "9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:51:00.595091 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:00.595073 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05-plugins-config-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:51:00.595136 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:00.595092 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/323ad275-3db9-43cf-96ea-ac33f6999051-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "323ad275-3db9-43cf-96ea-ac33f6999051" (UID: "323ad275-3db9-43cf-96ea-ac33f6999051"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:51:00.597023 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:00.597001 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323ad275-3db9-43cf-96ea-ac33f6999051-kube-api-access-q56l4" (OuterVolumeSpecName: "kube-api-access-q56l4") pod "323ad275-3db9-43cf-96ea-ac33f6999051" (UID: "323ad275-3db9-43cf-96ea-ac33f6999051"). InnerVolumeSpecName "kube-api-access-q56l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:51:00.597353 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:00.597336 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05-kube-api-access-7mqtx" (OuterVolumeSpecName: "kube-api-access-7mqtx") pod "9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05" (UID: "9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05"). InnerVolumeSpecName "kube-api-access-7mqtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:51:00.696650 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:00.696618 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/323ad275-3db9-43cf-96ea-ac33f6999051-plugins-config-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:51:00.696650 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:00.696647 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q56l4\" (UniqueName: \"kubernetes.io/projected/323ad275-3db9-43cf-96ea-ac33f6999051-kube-api-access-q56l4\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:51:00.696650 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:00.696656 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mqtx\" (UniqueName: \"kubernetes.io/projected/9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05-kube-api-access-7mqtx\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:51:01.067810 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:01.067774 2565 generic.go:356] "Generic (PLEG): container finished" podID="9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05" containerID="8737db05d6e2b08a9bd44c9e4a32e167d229e30ee15d1949b2bf24f4e27625d4" exitCode=0 Sep 05 06:51:01.068004 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:01.067823 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd" Sep 05 06:51:01.068004 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:01.067856 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd" event={"ID":"9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05","Type":"ContainerDied","Data":"8737db05d6e2b08a9bd44c9e4a32e167d229e30ee15d1949b2bf24f4e27625d4"} Sep 05 06:51:01.068004 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:01.067906 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd" event={"ID":"9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05","Type":"ContainerDied","Data":"12dae54960b2ab2c17ac9590b17796431a42a9f74350edb6be2f6e09545d6dfd"} Sep 05 06:51:01.068004 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:01.067928 2565 scope.go:117] "RemoveContainer" containerID="8737db05d6e2b08a9bd44c9e4a32e167d229e30ee15d1949b2bf24f4e27625d4" Sep 05 06:51:01.069411 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:01.069380 2565 generic.go:356] "Generic (PLEG): container finished" podID="323ad275-3db9-43cf-96ea-ac33f6999051" containerID="7938e76ffccd22d6d622ad1fcaa32e2e19750928940bafca178a5118abe1c657" exitCode=0 Sep 05 06:51:01.069530 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:01.069427 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8" Sep 05 06:51:01.069530 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:01.069446 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8" event={"ID":"323ad275-3db9-43cf-96ea-ac33f6999051","Type":"ContainerDied","Data":"7938e76ffccd22d6d622ad1fcaa32e2e19750928940bafca178a5118abe1c657"} Sep 05 06:51:01.069530 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:01.069485 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8" event={"ID":"323ad275-3db9-43cf-96ea-ac33f6999051","Type":"ContainerDied","Data":"dba3d5ab7f912fcd3c8757a8abaf087e16b3e154688b2cd7d4ae458390819dd4"} Sep 05 06:51:01.078493 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:01.078471 2565 scope.go:117] "RemoveContainer" containerID="8737db05d6e2b08a9bd44c9e4a32e167d229e30ee15d1949b2bf24f4e27625d4" Sep 05 06:51:01.078906 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:51:01.078783 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8737db05d6e2b08a9bd44c9e4a32e167d229e30ee15d1949b2bf24f4e27625d4\": container with ID starting with 8737db05d6e2b08a9bd44c9e4a32e167d229e30ee15d1949b2bf24f4e27625d4 not found: ID does not exist" containerID="8737db05d6e2b08a9bd44c9e4a32e167d229e30ee15d1949b2bf24f4e27625d4" Sep 05 06:51:01.078906 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:01.078837 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8737db05d6e2b08a9bd44c9e4a32e167d229e30ee15d1949b2bf24f4e27625d4"} err="failed to get container status \"8737db05d6e2b08a9bd44c9e4a32e167d229e30ee15d1949b2bf24f4e27625d4\": rpc error: code = NotFound desc = could not find container \"8737db05d6e2b08a9bd44c9e4a32e167d229e30ee15d1949b2bf24f4e27625d4\": container with ID starting with 8737db05d6e2b08a9bd44c9e4a32e167d229e30ee15d1949b2bf24f4e27625d4 not found: ID does not exist" Sep 05 06:51:01.078906 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:01.078859 2565 scope.go:117] "RemoveContainer" containerID="7938e76ffccd22d6d622ad1fcaa32e2e19750928940bafca178a5118abe1c657" Sep 05 06:51:01.087490 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:01.087469 2565 scope.go:117] "RemoveContainer" containerID="7938e76ffccd22d6d622ad1fcaa32e2e19750928940bafca178a5118abe1c657" Sep 05 06:51:01.087776 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:51:01.087759 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7938e76ffccd22d6d622ad1fcaa32e2e19750928940bafca178a5118abe1c657\": container with ID starting with 7938e76ffccd22d6d622ad1fcaa32e2e19750928940bafca178a5118abe1c657 not found: ID does not exist" containerID="7938e76ffccd22d6d622ad1fcaa32e2e19750928940bafca178a5118abe1c657" Sep 05 06:51:01.087840 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:01.087786 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7938e76ffccd22d6d622ad1fcaa32e2e19750928940bafca178a5118abe1c657"} err="failed to get container status \"7938e76ffccd22d6d622ad1fcaa32e2e19750928940bafca178a5118abe1c657\": rpc error: code = NotFound desc = could not find container \"7938e76ffccd22d6d622ad1fcaa32e2e19750928940bafca178a5118abe1c657\": container with ID starting with 7938e76ffccd22d6d622ad1fcaa32e2e19750928940bafca178a5118abe1c657 not found: ID does not exist" Sep 05 06:51:01.094068 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:01.094044 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8"] Sep 05 06:51:01.097758 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:01.097737 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9301f-d7b7869f4-nxqr8"] Sep 05 06:51:01.110319 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:01.110286 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd"] Sep 05 06:51:01.116388 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:01.116360 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9301f-65c4d46565-ldbqd"] Sep 05 06:51:02.166234 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:02.166200 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="323ad275-3db9-43cf-96ea-ac33f6999051" path="/var/lib/kubelet/pods/323ad275-3db9-43cf-96ea-ac33f6999051/volumes" Sep 05 06:51:02.166592 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:02.166584 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05" path="/var/lib/kubelet/pods/9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05/volumes" Sep 05 06:51:02.166862 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:02.166849 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9a1a7c4-6921-4ee8-843a-bcb71783db2e" path="/var/lib/kubelet/pods/b9a1a7c4-6921-4ee8-843a-bcb71783db2e/volumes" Sep 05 06:51:07.419366 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:07.419329 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-362c80-grgj5"] Sep 05 06:51:07.419753 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:07.419740 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="323ad275-3db9-43cf-96ea-ac33f6999051" containerName="payload-pre-processing" Sep 05 06:51:07.419753 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:07.419751 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="323ad275-3db9-43cf-96ea-ac33f6999051" containerName="payload-pre-processing" Sep 05 06:51:07.419831 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:07.419760 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9a1a7c4-6921-4ee8-843a-bcb71783db2e" containerName="revoke-keys" Sep 05 06:51:07.419831 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:07.419766 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9a1a7c4-6921-4ee8-843a-bcb71783db2e" containerName="revoke-keys" Sep 05 06:51:07.419831 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:07.419778 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05" containerName="payload-processing" Sep 05 06:51:07.419831 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:07.419784 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05" containerName="payload-processing" Sep 05 06:51:07.419945 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:07.419857 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f93a9a8-9bb5-4d12-ad80-f4d5dd8e8c05" containerName="payload-processing" Sep 05 06:51:07.419945 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:07.419866 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9a1a7c4-6921-4ee8-843a-bcb71783db2e" containerName="revoke-keys" Sep 05 06:51:07.419945 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:07.419874 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="323ad275-3db9-43cf-96ea-ac33f6999051" containerName="payload-pre-processing" Sep 05 06:51:07.424226 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:07.424211 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-362c80-grgj5" Sep 05 06:51:07.426413 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:07.426390 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slfqj\"" Sep 05 06:51:07.426673 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:07.426619 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 05 06:51:07.429505 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:07.429476 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-362c80-grgj5"] Sep 05 06:51:07.563022 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:07.562989 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18538ad2-92c5-4f3b-89ec-7523ecc75308-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-362c80-grgj5\" (UID: \"18538ad2-92c5-4f3b-89ec-7523ecc75308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-362c80-grgj5" Sep 05 06:51:07.664472 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:07.664434 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18538ad2-92c5-4f3b-89ec-7523ecc75308-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-362c80-grgj5\" (UID: \"18538ad2-92c5-4f3b-89ec-7523ecc75308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-362c80-grgj5" Sep 05 06:51:07.665279 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:07.665255 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18538ad2-92c5-4f3b-89ec-7523ecc75308-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-362c80-grgj5\" (UID: \"18538ad2-92c5-4f3b-89ec-7523ecc75308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-362c80-grgj5" Sep 05 06:51:07.736551 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:07.736473 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-362c80-grgj5" Sep 05 06:51:08.011405 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:08.011330 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-362c80-grgj5"] Sep 05 06:51:08.015180 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:51:08.015148 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18538ad2_92c5_4f3b_89ec_7523ecc75308.slice/crio-34f4991b39b60d5fe6a4c9d3bc7e694c3a167fd7df453f3ab48eb1b85e389d89 WatchSource:0}: Error finding container 34f4991b39b60d5fe6a4c9d3bc7e694c3a167fd7df453f3ab48eb1b85e389d89: Status 404 returned error can't find the container with id 34f4991b39b60d5fe6a4c9d3bc7e694c3a167fd7df453f3ab48eb1b85e389d89 Sep 05 06:51:08.116519 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:08.113155 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-362c80-grgj5" event={"ID":"18538ad2-92c5-4f3b-89ec-7523ecc75308","Type":"ContainerStarted","Data":"34f4991b39b60d5fe6a4c9d3bc7e694c3a167fd7df453f3ab48eb1b85e389d89"} Sep 05 06:51:09.120446 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:09.120415 2565 generic.go:356] "Generic (PLEG): container finished" podID="18538ad2-92c5-4f3b-89ec-7523ecc75308" containerID="c183d35a0aacb3d13df6a68bf04f870a61fcbfb4ca81ec8899dbbf88417fefe6" exitCode=0 Sep 05 06:51:09.120816 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:09.120506 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-362c80-grgj5" event={"ID":"18538ad2-92c5-4f3b-89ec-7523ecc75308","Type":"ContainerDied","Data":"c183d35a0aacb3d13df6a68bf04f870a61fcbfb4ca81ec8899dbbf88417fefe6"} Sep 05 06:51:10.264474 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:10.264453 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-362c80-grgj5" Sep 05 06:51:10.396134 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:10.396046 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18538ad2-92c5-4f3b-89ec-7523ecc75308-maas-api-service-ca\") pod \"18538ad2-92c5-4f3b-89ec-7523ecc75308\" (UID: \"18538ad2-92c5-4f3b-89ec-7523ecc75308\") " Sep 05 06:51:10.396423 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:10.396399 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18538ad2-92c5-4f3b-89ec-7523ecc75308-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "18538ad2-92c5-4f3b-89ec-7523ecc75308" (UID: "18538ad2-92c5-4f3b-89ec-7523ecc75308"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:51:10.497294 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:10.497263 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18538ad2-92c5-4f3b-89ec-7523ecc75308-maas-api-service-ca\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:51:11.132937 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:11.132900 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-362c80-grgj5" event={"ID":"18538ad2-92c5-4f3b-89ec-7523ecc75308","Type":"ContainerDied","Data":"34f4991b39b60d5fe6a4c9d3bc7e694c3a167fd7df453f3ab48eb1b85e389d89"} Sep 05 06:51:11.133085 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:11.132949 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34f4991b39b60d5fe6a4c9d3bc7e694c3a167fd7df453f3ab48eb1b85e389d89" Sep 05 06:51:11.133085 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:11.132910 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-362c80-grgj5" Sep 05 06:51:17.977145 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:17.977113 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph"] Sep 05 06:51:17.978391 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:17.977380 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph" podUID="29bb320e-e078-4212-9542-0640a6e2181e" containerName="payload-pre-processing" containerID="cri-o://0db0488f66313ccbe2179b52ddcef53a8049a7437fa358f0631ec033328312cc" gracePeriod=30 Sep 05 06:51:18.173394 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:18.173359 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-362c80-grgj5"] Sep 05 06:51:18.173564 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:18.173410 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-362c80-grgj5"] Sep 05 06:51:18.175543 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:18.175516 2565 generic.go:356] "Generic (PLEG): container finished" podID="29bb320e-e078-4212-9542-0640a6e2181e" containerID="0db0488f66313ccbe2179b52ddcef53a8049a7437fa358f0631ec033328312cc" exitCode=0 Sep 05 06:51:18.175678 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:18.175592 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph" event={"ID":"29bb320e-e078-4212-9542-0640a6e2181e","Type":"ContainerDied","Data":"0db0488f66313ccbe2179b52ddcef53a8049a7437fa358f0631ec033328312cc"} Sep 05 06:51:18.243380 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:18.242693 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph" Sep 05 06:51:18.372564 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:18.372530 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29bb320e-e078-4212-9542-0640a6e2181e-plugins-config-volume\") pod \"29bb320e-e078-4212-9542-0640a6e2181e\" (UID: \"29bb320e-e078-4212-9542-0640a6e2181e\") " Sep 05 06:51:18.372750 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:18.372593 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4m8g\" (UniqueName: \"kubernetes.io/projected/29bb320e-e078-4212-9542-0640a6e2181e-kube-api-access-z4m8g\") pod \"29bb320e-e078-4212-9542-0640a6e2181e\" (UID: \"29bb320e-e078-4212-9542-0640a6e2181e\") " Sep 05 06:51:18.372910 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:18.372885 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29bb320e-e078-4212-9542-0640a6e2181e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "29bb320e-e078-4212-9542-0640a6e2181e" (UID: "29bb320e-e078-4212-9542-0640a6e2181e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:51:18.374873 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:18.374850 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29bb320e-e078-4212-9542-0640a6e2181e-kube-api-access-z4m8g" (OuterVolumeSpecName: "kube-api-access-z4m8g") pod "29bb320e-e078-4212-9542-0640a6e2181e" (UID: "29bb320e-e078-4212-9542-0640a6e2181e"). InnerVolumeSpecName "kube-api-access-z4m8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:51:18.473749 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:18.473715 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29bb320e-e078-4212-9542-0640a6e2181e-plugins-config-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:51:18.473749 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:18.473745 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4m8g\" (UniqueName: \"kubernetes.io/projected/29bb320e-e078-4212-9542-0640a6e2181e-kube-api-access-z4m8g\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:51:19.181639 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:19.181611 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph" Sep 05 06:51:19.181639 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:19.181625 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph" event={"ID":"29bb320e-e078-4212-9542-0640a6e2181e","Type":"ContainerDied","Data":"b5a5a32e6834aae8cbad3f45c2424fba4b50d602e0d34c0c563834a166df3bd4"} Sep 05 06:51:19.182165 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:19.181678 2565 scope.go:117] "RemoveContainer" containerID="0db0488f66313ccbe2179b52ddcef53a8049a7437fa358f0631ec033328312cc" Sep 05 06:51:19.205832 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:19.205805 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph"] Sep 05 06:51:19.210344 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:19.210319 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-362c80-ddb887658-6drph"] Sep 05 06:51:20.166412 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:20.166379 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18538ad2-92c5-4f3b-89ec-7523ecc75308" path="/var/lib/kubelet/pods/18538ad2-92c5-4f3b-89ec-7523ecc75308/volumes" Sep 05 06:51:20.166765 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:20.166752 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29bb320e-e078-4212-9542-0640a6e2181e" path="/var/lib/kubelet/pods/29bb320e-e078-4212-9542-0640a6e2181e/volumes" Sep 05 06:51:25.021816 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:25.021776 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh"] Sep 05 06:51:25.022282 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:25.022048 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" podUID="f5557927-c8c3-4c86-a170-fe5b41cc23e8" containerName="istio-proxy" containerID="cri-o://361786cc72f60d15cecc92c6569485e9496f71820399f55ddfd58a86f2b06042" gracePeriod=30 Sep 05 06:51:25.408421 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:25.408385 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e47e8-b9ct8"] Sep 05 06:51:25.408804 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:25.408792 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29bb320e-e078-4212-9542-0640a6e2181e" containerName="payload-pre-processing" Sep 05 06:51:25.408851 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:25.408806 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="29bb320e-e078-4212-9542-0640a6e2181e" containerName="payload-pre-processing" Sep 05 06:51:25.408851 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:25.408827 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18538ad2-92c5-4f3b-89ec-7523ecc75308" containerName="revoke-keys" Sep 05 06:51:25.408851 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:25.408833 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="18538ad2-92c5-4f3b-89ec-7523ecc75308" containerName="revoke-keys" Sep 05 06:51:25.408946 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:25.408896 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="18538ad2-92c5-4f3b-89ec-7523ecc75308" containerName="revoke-keys" Sep 05 06:51:25.408946 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:25.408905 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="29bb320e-e078-4212-9542-0640a6e2181e" containerName="payload-pre-processing" Sep 05 06:51:25.413060 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:25.413040 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e47e8-b9ct8" Sep 05 06:51:25.415881 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:25.415862 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 05 06:51:25.419931 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:25.419915 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slfqj\"" Sep 05 06:51:25.424972 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:25.424949 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e47e8-b9ct8"] Sep 05 06:51:25.441326 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:25.441284 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbe9a007-921c-479b-b9c8-abf5351fa5de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5e47e8-b9ct8\" (UID: \"dbe9a007-921c-479b-b9c8-abf5351fa5de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e47e8-b9ct8" Sep 05 06:51:25.541745 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:25.541714 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbe9a007-921c-479b-b9c8-abf5351fa5de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5e47e8-b9ct8\" (UID: \"dbe9a007-921c-479b-b9c8-abf5351fa5de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e47e8-b9ct8" Sep 05 06:51:25.542292 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:25.542273 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbe9a007-921c-479b-b9c8-abf5351fa5de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5e47e8-b9ct8\" (UID: \"dbe9a007-921c-479b-b9c8-abf5351fa5de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e47e8-b9ct8" Sep 05 06:51:25.723608 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:25.723513 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e47e8-b9ct8" Sep 05 06:51:26.003368 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:26.003338 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e47e8-b9ct8"] Sep 05 06:51:26.005801 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:51:26.005775 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe9a007_921c_479b_b9c8_abf5351fa5de.slice/crio-ef20b6debe120bbe2aaa41b591e098988051fb53c0fb205e6dfe7549218cdf7b WatchSource:0}: Error finding container ef20b6debe120bbe2aaa41b591e098988051fb53c0fb205e6dfe7549218cdf7b: Status 404 returned error can't find the container with id ef20b6debe120bbe2aaa41b591e098988051fb53c0fb205e6dfe7549218cdf7b Sep 05 06:51:26.221110 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:26.221070 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e47e8-b9ct8" event={"ID":"dbe9a007-921c-479b-b9c8-abf5351fa5de","Type":"ContainerStarted","Data":"ef20b6debe120bbe2aaa41b591e098988051fb53c0fb205e6dfe7549218cdf7b"} Sep 05 06:51:27.228367 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:27.228331 2565 generic.go:356] "Generic (PLEG): container finished" podID="dbe9a007-921c-479b-b9c8-abf5351fa5de" containerID="9bc0f56ab8bcd7e491f4d94f5d51e201bdc863c80efa8fd03b14d816e950e0f6" exitCode=0 Sep 05 06:51:27.228744 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:27.228379 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e47e8-b9ct8" event={"ID":"dbe9a007-921c-479b-b9c8-abf5351fa5de","Type":"ContainerDied","Data":"9bc0f56ab8bcd7e491f4d94f5d51e201bdc863c80efa8fd03b14d816e950e0f6"} Sep 05 06:51:28.374375 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:28.374342 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e47e8-b9ct8" Sep 05 06:51:28.466288 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:28.466240 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbe9a007-921c-479b-b9c8-abf5351fa5de-maas-api-service-ca\") pod \"dbe9a007-921c-479b-b9c8-abf5351fa5de\" (UID: \"dbe9a007-921c-479b-b9c8-abf5351fa5de\") " Sep 05 06:51:28.466618 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:28.466593 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbe9a007-921c-479b-b9c8-abf5351fa5de-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dbe9a007-921c-479b-b9c8-abf5351fa5de" (UID: "dbe9a007-921c-479b-b9c8-abf5351fa5de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:51:28.567777 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:28.567695 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbe9a007-921c-479b-b9c8-abf5351fa5de-maas-api-service-ca\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:51:29.239714 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:29.239682 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e47e8-b9ct8" Sep 05 06:51:29.239892 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:29.239716 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e47e8-b9ct8" event={"ID":"dbe9a007-921c-479b-b9c8-abf5351fa5de","Type":"ContainerDied","Data":"ef20b6debe120bbe2aaa41b591e098988051fb53c0fb205e6dfe7549218cdf7b"} Sep 05 06:51:29.239892 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:29.239769 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef20b6debe120bbe2aaa41b591e098988051fb53c0fb205e6dfe7549218cdf7b" Sep 05 06:51:30.247456 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.247422 2565 generic.go:356] "Generic (PLEG): container finished" podID="f5557927-c8c3-4c86-a170-fe5b41cc23e8" containerID="361786cc72f60d15cecc92c6569485e9496f71820399f55ddfd58a86f2b06042" exitCode=0 Sep 05 06:51:30.247852 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.247481 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" event={"ID":"f5557927-c8c3-4c86-a170-fe5b41cc23e8","Type":"ContainerDied","Data":"361786cc72f60d15cecc92c6569485e9496f71820399f55ddfd58a86f2b06042"} Sep 05 06:51:30.247852 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.247544 2565 scope.go:117] "RemoveContainer" containerID="d0c44356d7cea0f848ce1e1c3b97959dcba6433beb78cf104206d8b9ce50cf38" Sep 05 06:51:30.283708 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.283681 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:51:30.384617 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.384581 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-workload-certs\") pod \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " Sep 05 06:51:30.384816 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.384627 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istio-envoy\") pod \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " Sep 05 06:51:30.384816 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.384657 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-credential-socket\") pod \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " Sep 05 06:51:30.384816 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.384678 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-workload-socket\") pod \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " Sep 05 06:51:30.384816 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.384708 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istio-token\") pod \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " Sep 05 06:51:30.384816 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.384764 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istio-podinfo\") pod \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " Sep 05 06:51:30.384816 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.384793 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istio-data\") pod \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " Sep 05 06:51:30.385132 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.384846 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istiod-ca-cert\") pod \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " Sep 05 06:51:30.385132 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.384902 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n64vl\" (UniqueName: \"kubernetes.io/projected/f5557927-c8c3-4c86-a170-fe5b41cc23e8-kube-api-access-n64vl\") pod \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\" (UID: \"f5557927-c8c3-4c86-a170-fe5b41cc23e8\") " Sep 05 06:51:30.385132 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.384908 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f5557927-c8c3-4c86-a170-fe5b41cc23e8" (UID: "f5557927-c8c3-4c86-a170-fe5b41cc23e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:51:30.385132 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.384981 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f5557927-c8c3-4c86-a170-fe5b41cc23e8" (UID: "f5557927-c8c3-4c86-a170-fe5b41cc23e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:51:30.385389 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.385135 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f5557927-c8c3-4c86-a170-fe5b41cc23e8" (UID: "f5557927-c8c3-4c86-a170-fe5b41cc23e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:51:30.385389 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.385163 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istio-data" (OuterVolumeSpecName: "istio-data") pod "f5557927-c8c3-4c86-a170-fe5b41cc23e8" (UID: "f5557927-c8c3-4c86-a170-fe5b41cc23e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:51:30.385389 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.385364 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f5557927-c8c3-4c86-a170-fe5b41cc23e8" (UID: "f5557927-c8c3-4c86-a170-fe5b41cc23e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:51:30.385543 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.385446 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istiod-ca-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:51:30.385543 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.385469 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-workload-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:51:30.385543 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.385484 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-credential-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:51:30.385665 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.385561 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-workload-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:51:30.385706 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.385667 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istio-data\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:51:30.387130 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.387095 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f5557927-c8c3-4c86-a170-fe5b41cc23e8" (UID: "f5557927-c8c3-4c86-a170-fe5b41cc23e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 05 06:51:30.387244 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.387127 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istio-token" (OuterVolumeSpecName: "istio-token") pod "f5557927-c8c3-4c86-a170-fe5b41cc23e8" (UID: "f5557927-c8c3-4c86-a170-fe5b41cc23e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:51:30.387553 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.387530 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f5557927-c8c3-4c86-a170-fe5b41cc23e8" (UID: "f5557927-c8c3-4c86-a170-fe5b41cc23e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:51:30.387686 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.387671 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5557927-c8c3-4c86-a170-fe5b41cc23e8-kube-api-access-n64vl" (OuterVolumeSpecName: "kube-api-access-n64vl") pod "f5557927-c8c3-4c86-a170-fe5b41cc23e8" (UID: "f5557927-c8c3-4c86-a170-fe5b41cc23e8"). InnerVolumeSpecName "kube-api-access-n64vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:51:30.487119 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.487087 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istio-token\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:51:30.487119 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.487116 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istio-podinfo\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:51:30.487119 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.487126 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n64vl\" (UniqueName: \"kubernetes.io/projected/f5557927-c8c3-4c86-a170-fe5b41cc23e8-kube-api-access-n64vl\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:51:30.487391 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:30.487134 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5557927-c8c3-4c86-a170-fe5b41cc23e8-istio-envoy\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:51:31.254063 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:31.254030 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" Sep 05 06:51:31.254577 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:31.254068 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh" event={"ID":"f5557927-c8c3-4c86-a170-fe5b41cc23e8","Type":"ContainerDied","Data":"6cf64688149875075871f9508056670a23512aea5c9b5495b0ce4a61cfcbbb34"} Sep 05 06:51:31.254577 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:31.254126 2565 scope.go:117] "RemoveContainer" containerID="361786cc72f60d15cecc92c6569485e9496f71820399f55ddfd58a86f2b06042" Sep 05 06:51:31.287751 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:31.287717 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh"] Sep 05 06:51:31.300350 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:31.300304 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-362c80-openshift-default-7c985b7574-jksqh"] Sep 05 06:51:32.166426 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:32.166396 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5557927-c8c3-4c86-a170-fe5b41cc23e8" path="/var/lib/kubelet/pods/f5557927-c8c3-4c86-a170-fe5b41cc23e8/volumes" Sep 05 06:51:34.166445 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.166420 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx"] Sep 05 06:51:34.166858 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.166845 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbe9a007-921c-479b-b9c8-abf5351fa5de" containerName="revoke-keys" Sep 05 06:51:34.166915 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.166861 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbe9a007-921c-479b-b9c8-abf5351fa5de" containerName="revoke-keys" Sep 05 06:51:34.166915 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.166876 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5557927-c8c3-4c86-a170-fe5b41cc23e8" containerName="istio-proxy" Sep 05 06:51:34.166915 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.166885 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5557927-c8c3-4c86-a170-fe5b41cc23e8" containerName="istio-proxy" Sep 05 06:51:34.166915 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.166893 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5557927-c8c3-4c86-a170-fe5b41cc23e8" containerName="istio-proxy" Sep 05 06:51:34.166915 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.166898 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5557927-c8c3-4c86-a170-fe5b41cc23e8" containerName="istio-proxy" Sep 05 06:51:34.167089 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.166977 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5557927-c8c3-4c86-a170-fe5b41cc23e8" containerName="istio-proxy" Sep 05 06:51:34.167089 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.166987 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbe9a007-921c-479b-b9c8-abf5351fa5de" containerName="revoke-keys" Sep 05 06:51:34.167089 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.166993 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5557927-c8c3-4c86-a170-fe5b41cc23e8" containerName="istio-proxy" Sep 05 06:51:34.171600 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.171583 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.173898 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.173869 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-d36c73-openshift-default-dockercfg-zzlbv\"" Sep 05 06:51:34.176974 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.176954 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx"] Sep 05 06:51:34.219681 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.219646 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istiod-ca-cert\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.219833 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.219689 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istio-token\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.219833 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.219726 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-workload-socket\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.219833 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.219790 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th4sz\" (UniqueName: \"kubernetes.io/projected/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-kube-api-access-th4sz\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.219981 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.219855 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-credential-socket\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.219981 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.219882 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-workload-certs\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.219981 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.219969 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istio-data\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.220135 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.219999 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istio-podinfo\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.220135 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.220031 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istio-envoy\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.321282 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.321243 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th4sz\" (UniqueName: \"kubernetes.io/projected/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-kube-api-access-th4sz\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.321503 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.321325 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-credential-socket\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.321503 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.321348 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-workload-certs\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.321503 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.321403 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istio-data\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.321503 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.321442 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istio-podinfo\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.321710 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.321576 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istio-envoy\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.321710 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.321630 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istiod-ca-cert\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.321710 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.321660 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istio-token\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.321710 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.321697 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-workload-socket\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.321873 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.321778 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-credential-socket\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.322035 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.322009 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-workload-certs\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.322161 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.322094 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istio-data\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.322386 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.322366 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-workload-socket\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.322775 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.322752 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istiod-ca-cert\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.323988 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.323960 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istio-envoy\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.324171 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.324154 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istio-podinfo\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.329668 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.329645 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istio-token\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.329959 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.329944 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th4sz\" (UniqueName: \"kubernetes.io/projected/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-kube-api-access-th4sz\") pod \"e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.484334 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.484226 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:34.766182 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.766157 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx"] Sep 05 06:51:34.768069 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:51:34.768044 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36796a4_e4e2_4c53_83e6_0e7b56f3d854.slice/crio-9dc2a9ce7666b37607116e7042eb52c8e3d339306f965b635acdef49c113f360 WatchSource:0}: Error finding container 9dc2a9ce7666b37607116e7042eb52c8e3d339306f965b635acdef49c113f360: Status 404 returned error can't find the container with id 9dc2a9ce7666b37607116e7042eb52c8e3d339306f965b635acdef49c113f360 Sep 05 06:51:34.770292 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.770264 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:51:34.770395 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.770361 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:51:34.770437 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:34.770391 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:51:35.280185 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:35.280149 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" event={"ID":"d36796a4-e4e2-4c53-83e6-0e7b56f3d854","Type":"ContainerStarted","Data":"1396c5e65f0c2bb79fa844a754a2797439a37f9ec26bb61525de25ba4758f007"} Sep 05 06:51:35.280639 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:35.280218 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" event={"ID":"d36796a4-e4e2-4c53-83e6-0e7b56f3d854","Type":"ContainerStarted","Data":"9dc2a9ce7666b37607116e7042eb52c8e3d339306f965b635acdef49c113f360"} Sep 05 06:51:35.301790 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:35.301741 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" podStartSLOduration=1.301720235 podStartE2EDuration="1.301720235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:51:35.300481193 +0000 UTC m=+1519.685024328" watchObservedRunningTime="2026-09-05 06:51:35.301720235 +0000 UTC m=+1519.686263415" Sep 05 06:51:35.484890 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:35.484858 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:35.489941 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:35.489918 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:36.102795 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.102758 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh"] Sep 05 06:51:36.103052 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.103027 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh" podUID="7a24b1e8-87a7-4106-b294-e4d0f6b75a0f" containerName="payload-processing" containerID="cri-o://a4e85180fca7041c527543e45390fe66de49092c524c1f05e70383814376d336" gracePeriod=30 Sep 05 06:51:36.111618 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.111589 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d"] Sep 05 06:51:36.111832 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.111806 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d" podUID="11dde2b8-22a2-4bcb-aa75-65d15139c39e" containerName="payload-pre-processing" containerID="cri-o://e785aa9b25ea882250609a035005a3cd1c8d7f1512bbb13b38c8b01a1d06934f" gracePeriod=30 Sep 05 06:51:36.288951 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.288920 2565 generic.go:356] "Generic (PLEG): container finished" podID="7a24b1e8-87a7-4106-b294-e4d0f6b75a0f" containerID="a4e85180fca7041c527543e45390fe66de49092c524c1f05e70383814376d336" exitCode=0 Sep 05 06:51:36.289303 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.288986 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh" event={"ID":"7a24b1e8-87a7-4106-b294-e4d0f6b75a0f","Type":"ContainerDied","Data":"a4e85180fca7041c527543e45390fe66de49092c524c1f05e70383814376d336"} Sep 05 06:51:36.290513 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.290492 2565 generic.go:356] "Generic (PLEG): container finished" podID="11dde2b8-22a2-4bcb-aa75-65d15139c39e" containerID="e785aa9b25ea882250609a035005a3cd1c8d7f1512bbb13b38c8b01a1d06934f" exitCode=0 Sep 05 06:51:36.290648 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.290532 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d" event={"ID":"11dde2b8-22a2-4bcb-aa75-65d15139c39e","Type":"ContainerDied","Data":"e785aa9b25ea882250609a035005a3cd1c8d7f1512bbb13b38c8b01a1d06934f"} Sep 05 06:51:36.291343 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.291008 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:36.292215 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.292189 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:36.306393 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.306372 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e47e8-b9ct8"] Sep 05 06:51:36.308302 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.308281 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e47e8-b9ct8"] Sep 05 06:51:36.411161 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.411137 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh" Sep 05 06:51:36.442998 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.442973 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d" Sep 05 06:51:36.447072 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.446869 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a24b1e8-87a7-4106-b294-e4d0f6b75a0f-plugins-config-volume\") pod \"7a24b1e8-87a7-4106-b294-e4d0f6b75a0f\" (UID: \"7a24b1e8-87a7-4106-b294-e4d0f6b75a0f\") " Sep 05 06:51:36.447072 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.446948 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjgc5\" (UniqueName: \"kubernetes.io/projected/7a24b1e8-87a7-4106-b294-e4d0f6b75a0f-kube-api-access-xjgc5\") pod \"7a24b1e8-87a7-4106-b294-e4d0f6b75a0f\" (UID: \"7a24b1e8-87a7-4106-b294-e4d0f6b75a0f\") " Sep 05 06:51:36.447352 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.447260 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a24b1e8-87a7-4106-b294-e4d0f6b75a0f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7a24b1e8-87a7-4106-b294-e4d0f6b75a0f" (UID: "7a24b1e8-87a7-4106-b294-e4d0f6b75a0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:51:36.447799 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.447765 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a24b1e8-87a7-4106-b294-e4d0f6b75a0f-plugins-config-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:51:36.450551 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.450476 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a24b1e8-87a7-4106-b294-e4d0f6b75a0f-kube-api-access-xjgc5" (OuterVolumeSpecName: "kube-api-access-xjgc5") pod "7a24b1e8-87a7-4106-b294-e4d0f6b75a0f" (UID: "7a24b1e8-87a7-4106-b294-e4d0f6b75a0f"). InnerVolumeSpecName "kube-api-access-xjgc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:51:36.548829 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.548762 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmsz4\" (UniqueName: \"kubernetes.io/projected/11dde2b8-22a2-4bcb-aa75-65d15139c39e-kube-api-access-qmsz4\") pod \"11dde2b8-22a2-4bcb-aa75-65d15139c39e\" (UID: \"11dde2b8-22a2-4bcb-aa75-65d15139c39e\") " Sep 05 06:51:36.548975 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.548893 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11dde2b8-22a2-4bcb-aa75-65d15139c39e-plugins-config-volume\") pod \"11dde2b8-22a2-4bcb-aa75-65d15139c39e\" (UID: \"11dde2b8-22a2-4bcb-aa75-65d15139c39e\") " Sep 05 06:51:36.549178 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.549150 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjgc5\" (UniqueName: \"kubernetes.io/projected/7a24b1e8-87a7-4106-b294-e4d0f6b75a0f-kube-api-access-xjgc5\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:51:36.549255 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.549219 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11dde2b8-22a2-4bcb-aa75-65d15139c39e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "11dde2b8-22a2-4bcb-aa75-65d15139c39e" (UID: "11dde2b8-22a2-4bcb-aa75-65d15139c39e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:51:36.551008 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.550988 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11dde2b8-22a2-4bcb-aa75-65d15139c39e-kube-api-access-qmsz4" (OuterVolumeSpecName: "kube-api-access-qmsz4") pod "11dde2b8-22a2-4bcb-aa75-65d15139c39e" (UID: "11dde2b8-22a2-4bcb-aa75-65d15139c39e"). InnerVolumeSpecName "kube-api-access-qmsz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:51:36.650517 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.650442 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11dde2b8-22a2-4bcb-aa75-65d15139c39e-plugins-config-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:51:36.650517 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:36.650465 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmsz4\" (UniqueName: \"kubernetes.io/projected/11dde2b8-22a2-4bcb-aa75-65d15139c39e-kube-api-access-qmsz4\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:51:37.297593 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:37.297565 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh" Sep 05 06:51:37.298061 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:37.297602 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh" event={"ID":"7a24b1e8-87a7-4106-b294-e4d0f6b75a0f","Type":"ContainerDied","Data":"744265594f425e59f60dbc83f3b293c31bfa6d3368b479eb0bd8f2b5160faf6d"} Sep 05 06:51:37.298061 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:37.297677 2565 scope.go:117] "RemoveContainer" containerID="a4e85180fca7041c527543e45390fe66de49092c524c1f05e70383814376d336" Sep 05 06:51:37.299847 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:37.299809 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d" Sep 05 06:51:37.300521 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:37.300396 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d" event={"ID":"11dde2b8-22a2-4bcb-aa75-65d15139c39e","Type":"ContainerDied","Data":"e76c33b47ab670bebc63af971949195fb764390c7a1281dec6ad928fc28ab937"} Sep 05 06:51:37.309825 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:37.309809 2565 scope.go:117] "RemoveContainer" containerID="e785aa9b25ea882250609a035005a3cd1c8d7f1512bbb13b38c8b01a1d06934f" Sep 05 06:51:37.324087 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:37.324067 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh"] Sep 05 06:51:37.334397 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:37.334374 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5e47e8-5d46489b75-sf7jh"] Sep 05 06:51:37.345380 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:37.345355 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d"] Sep 05 06:51:37.350004 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:37.349984 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e47e8-84774b79db-rlf5d"] Sep 05 06:51:38.165498 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:38.165460 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11dde2b8-22a2-4bcb-aa75-65d15139c39e" path="/var/lib/kubelet/pods/11dde2b8-22a2-4bcb-aa75-65d15139c39e/volumes" Sep 05 06:51:38.166013 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:38.165997 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a24b1e8-87a7-4106-b294-e4d0f6b75a0f" path="/var/lib/kubelet/pods/7a24b1e8-87a7-4106-b294-e4d0f6b75a0f/volumes" Sep 05 06:51:38.166297 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:38.166285 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbe9a007-921c-479b-b9c8-abf5351fa5de" path="/var/lib/kubelet/pods/dbe9a007-921c-479b-b9c8-abf5351fa5de/volumes" Sep 05 06:51:39.165392 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:39.165355 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd"] Sep 05 06:51:39.165778 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:39.165765 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11dde2b8-22a2-4bcb-aa75-65d15139c39e" containerName="payload-pre-processing" Sep 05 06:51:39.165837 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:39.165781 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="11dde2b8-22a2-4bcb-aa75-65d15139c39e" containerName="payload-pre-processing" Sep 05 06:51:39.165837 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:39.165793 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a24b1e8-87a7-4106-b294-e4d0f6b75a0f" containerName="payload-processing" Sep 05 06:51:39.165837 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:39.165799 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a24b1e8-87a7-4106-b294-e4d0f6b75a0f" containerName="payload-processing" Sep 05 06:51:39.165929 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:39.165910 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a24b1e8-87a7-4106-b294-e4d0f6b75a0f" containerName="payload-processing" Sep 05 06:51:39.165964 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:39.165929 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="11dde2b8-22a2-4bcb-aa75-65d15139c39e" containerName="payload-pre-processing" Sep 05 06:51:39.170563 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:39.170543 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd" Sep 05 06:51:39.172885 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:39.172865 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d36c73-dockercfg-g8cbv\"" Sep 05 06:51:39.173213 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:39.173200 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d36c73\"" Sep 05 06:51:39.179189 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:39.179158 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd"] Sep 05 06:51:39.272049 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:39.272018 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39c3e90f-d3a3-450b-8476-90eed1d927ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd\" (UID: \"39c3e90f-d3a3-450b-8476-90eed1d927ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd" Sep 05 06:51:39.272211 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:39.272173 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlz8r\" (UniqueName: \"kubernetes.io/projected/39c3e90f-d3a3-450b-8476-90eed1d927ed-kube-api-access-tlz8r\") pod \"payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd\" (UID: \"39c3e90f-d3a3-450b-8476-90eed1d927ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd" Sep 05 06:51:39.373731 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:39.373686 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlz8r\" (UniqueName: \"kubernetes.io/projected/39c3e90f-d3a3-450b-8476-90eed1d927ed-kube-api-access-tlz8r\") pod \"payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd\" (UID: \"39c3e90f-d3a3-450b-8476-90eed1d927ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd" Sep 05 06:51:39.373895 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:39.373777 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39c3e90f-d3a3-450b-8476-90eed1d927ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd\" (UID: \"39c3e90f-d3a3-450b-8476-90eed1d927ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd" Sep 05 06:51:39.374536 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:39.374516 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39c3e90f-d3a3-450b-8476-90eed1d927ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd\" (UID: \"39c3e90f-d3a3-450b-8476-90eed1d927ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd" Sep 05 06:51:39.382681 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:39.382656 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlz8r\" (UniqueName: \"kubernetes.io/projected/39c3e90f-d3a3-450b-8476-90eed1d927ed-kube-api-access-tlz8r\") pod \"payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd\" (UID: \"39c3e90f-d3a3-450b-8476-90eed1d927ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd" Sep 05 06:51:39.482520 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:39.482425 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd" Sep 05 06:51:39.759236 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:39.759176 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd"] Sep 05 06:51:39.761467 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:51:39.761438 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c3e90f_d3a3_450b_8476_90eed1d927ed.slice/crio-46905ddd803a022d46dc6aab85f4053062f699932226f37a9936f8b81db470de WatchSource:0}: Error finding container 46905ddd803a022d46dc6aab85f4053062f699932226f37a9936f8b81db470de: Status 404 returned error can't find the container with id 46905ddd803a022d46dc6aab85f4053062f699932226f37a9936f8b81db470de Sep 05 06:51:40.323476 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:40.323431 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd" event={"ID":"39c3e90f-d3a3-450b-8476-90eed1d927ed","Type":"ContainerStarted","Data":"f5c87a86feb73ea22e768ba5678a0a27b398a02db6dc5ef622b1575b27308a6f"} Sep 05 06:51:40.323865 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:40.323494 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd" event={"ID":"39c3e90f-d3a3-450b-8476-90eed1d927ed","Type":"ContainerStarted","Data":"46905ddd803a022d46dc6aab85f4053062f699932226f37a9936f8b81db470de"} Sep 05 06:51:40.323865 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:40.323544 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd" Sep 05 06:51:40.345301 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:40.345256 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd" podStartSLOduration=1.345241676 podStartE2EDuration="1.345241676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:51:40.343014671 +0000 UTC m=+1524.727557787" watchObservedRunningTime="2026-09-05 06:51:40.345241676 +0000 UTC m=+1524.729784800" Sep 05 06:51:51.331381 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:51.331343 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd" Sep 05 06:51:56.429242 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:56.429161 2565 generic.go:356] "Generic (PLEG): container finished" podID="d36796a4-e4e2-4c53-83e6-0e7b56f3d854" containerID="1396c5e65f0c2bb79fa844a754a2797439a37f9ec26bb61525de25ba4758f007" exitCode=0 Sep 05 06:51:56.429677 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:56.429248 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" event={"ID":"d36796a4-e4e2-4c53-83e6-0e7b56f3d854","Type":"ContainerDied","Data":"1396c5e65f0c2bb79fa844a754a2797439a37f9ec26bb61525de25ba4758f007"} Sep 05 06:51:56.429724 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:56.429690 2565 scope.go:117] "RemoveContainer" containerID="1396c5e65f0c2bb79fa844a754a2797439a37f9ec26bb61525de25ba4758f007" Sep 05 06:51:56.430874 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:56.430850 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:51:56.430929 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:56.430916 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:51:56.430964 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:56.430945 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 05 06:51:57.436656 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:57.436623 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" event={"ID":"d36796a4-e4e2-4c53-83e6-0e7b56f3d854","Type":"ContainerStarted","Data":"123904d04e299aa8cce374bdf1857b16bb964fc52ed201d6caa1a38cbc87c8e0"} Sep 05 06:51:57.487499 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:57.487454 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:51:57.487913 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:57.487881 2565 patch_prober.go:28] interesting pod/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Sep 05 06:51:57.488022 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:57.487944 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" podUID="d36796a4-e4e2-4c53-83e6-0e7b56f3d854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Sep 05 06:51:58.484890 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:58.484853 2565 patch_prober.go:28] interesting pod/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Sep 05 06:51:58.485509 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:58.485477 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" podUID="d36796a4-e4e2-4c53-83e6-0e7b56f3d854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Sep 05 06:51:59.484744 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:59.484708 2565 patch_prober.go:28] interesting pod/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Sep 05 06:51:59.485024 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:51:59.484997 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" podUID="d36796a4-e4e2-4c53-83e6-0e7b56f3d854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Sep 05 06:52:00.484894 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:00.484847 2565 patch_prober.go:28] interesting pod/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Sep 05 06:52:00.485101 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:00.484917 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" podUID="d36796a4-e4e2-4c53-83e6-0e7b56f3d854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Sep 05 06:52:01.484739 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:01.484697 2565 patch_prober.go:28] interesting pod/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Sep 05 06:52:01.484954 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:01.484788 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" podUID="d36796a4-e4e2-4c53-83e6-0e7b56f3d854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Sep 05 06:52:02.484839 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:02.484803 2565 patch_prober.go:28] interesting pod/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Sep 05 06:52:02.485365 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:02.484866 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" podUID="d36796a4-e4e2-4c53-83e6-0e7b56f3d854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Sep 05 06:52:03.489924 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:03.489892 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:52:03.490442 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:03.490334 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:52:03.490954 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:03.490935 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:52:08.747727 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:08.747683 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5"] Sep 05 06:52:08.751471 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:08.751448 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5" Sep 05 06:52:08.754210 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:08.754183 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b49086-dockercfg-ndrs8\"" Sep 05 06:52:08.754347 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:08.754192 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b49086\"" Sep 05 06:52:08.762632 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:08.762600 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5"] Sep 05 06:52:08.765666 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:08.765640 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl"] Sep 05 06:52:08.768784 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:08.768766 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl" Sep 05 06:52:08.774840 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:08.774813 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5\" (UID: \"13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5" Sep 05 06:52:08.774959 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:08.774935 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gr52\" (UniqueName: \"kubernetes.io/projected/13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60-kube-api-access-6gr52\") pod \"payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5\" (UID: \"13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5" Sep 05 06:52:08.781996 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:08.781965 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl"] Sep 05 06:52:08.875664 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:08.875633 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5\" (UID: \"13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5" Sep 05 06:52:08.875861 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:08.875700 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gr52\" (UniqueName: \"kubernetes.io/projected/13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60-kube-api-access-6gr52\") pod \"payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5\" (UID: \"13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5" Sep 05 06:52:08.875861 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:08.875723 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b2d651a-289f-4daf-918c-3a58067ebb70-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl\" (UID: \"0b2d651a-289f-4daf-918c-3a58067ebb70\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl" Sep 05 06:52:08.875969 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:08.875887 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jcvs\" (UniqueName: \"kubernetes.io/projected/0b2d651a-289f-4daf-918c-3a58067ebb70-kube-api-access-9jcvs\") pod \"payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl\" (UID: \"0b2d651a-289f-4daf-918c-3a58067ebb70\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl" Sep 05 06:52:08.876425 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:08.876406 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5\" (UID: \"13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5" Sep 05 06:52:08.889569 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:08.889548 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gr52\" (UniqueName: \"kubernetes.io/projected/13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60-kube-api-access-6gr52\") pod \"payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5\" (UID: \"13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5" Sep 05 06:52:08.976961 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:08.976914 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b2d651a-289f-4daf-918c-3a58067ebb70-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl\" (UID: \"0b2d651a-289f-4daf-918c-3a58067ebb70\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl" Sep 05 06:52:08.977167 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:08.977013 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jcvs\" (UniqueName: \"kubernetes.io/projected/0b2d651a-289f-4daf-918c-3a58067ebb70-kube-api-access-9jcvs\") pod \"payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl\" (UID: \"0b2d651a-289f-4daf-918c-3a58067ebb70\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl" Sep 05 06:52:08.977991 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:08.977962 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b2d651a-289f-4daf-918c-3a58067ebb70-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl\" (UID: \"0b2d651a-289f-4daf-918c-3a58067ebb70\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl" Sep 05 06:52:08.989365 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:08.989334 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jcvs\" (UniqueName: \"kubernetes.io/projected/0b2d651a-289f-4daf-918c-3a58067ebb70-kube-api-access-9jcvs\") pod \"payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl\" (UID: \"0b2d651a-289f-4daf-918c-3a58067ebb70\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl" Sep 05 06:52:09.062567 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:09.062478 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5" Sep 05 06:52:09.079877 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:09.079849 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl" Sep 05 06:52:09.343531 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:09.343501 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5"] Sep 05 06:52:09.345647 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:52:09.345617 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a9ec1f_f301_4cc9_9e8d_05cf09ee4e60.slice/crio-3cd1a373814539426ed2dd6292e1ced8a9c874f362e1406d27a280f8fbd55cd1 WatchSource:0}: Error finding container 3cd1a373814539426ed2dd6292e1ced8a9c874f362e1406d27a280f8fbd55cd1: Status 404 returned error can't find the container with id 3cd1a373814539426ed2dd6292e1ced8a9c874f362e1406d27a280f8fbd55cd1 Sep 05 06:52:09.366649 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:09.366623 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl"] Sep 05 06:52:09.368414 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:52:09.368391 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b2d651a_289f_4daf_918c_3a58067ebb70.slice/crio-c151a60415cf74c622a59b02e99fbc718badec1a4293d9ade71e1b19a80bc153 WatchSource:0}: Error finding container c151a60415cf74c622a59b02e99fbc718badec1a4293d9ade71e1b19a80bc153: Status 404 returned error can't find the container with id c151a60415cf74c622a59b02e99fbc718badec1a4293d9ade71e1b19a80bc153 Sep 05 06:52:09.505446 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:09.505413 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5" event={"ID":"13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60","Type":"ContainerStarted","Data":"edbade95fc902b9c1998ea28107a50b91c142f3182486f0a5cd93c052690c2d8"} Sep 05 06:52:09.505625 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:09.505468 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5" event={"ID":"13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60","Type":"ContainerStarted","Data":"3cd1a373814539426ed2dd6292e1ced8a9c874f362e1406d27a280f8fbd55cd1"} Sep 05 06:52:09.505625 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:09.505528 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5" Sep 05 06:52:09.507366 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:09.507342 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl" event={"ID":"0b2d651a-289f-4daf-918c-3a58067ebb70","Type":"ContainerStarted","Data":"179fd9400b565781ec3d8c37dd2275123264643dd1c1727cf4d82ab2e4d49286"} Sep 05 06:52:09.507478 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:09.507382 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl" event={"ID":"0b2d651a-289f-4daf-918c-3a58067ebb70","Type":"ContainerStarted","Data":"c151a60415cf74c622a59b02e99fbc718badec1a4293d9ade71e1b19a80bc153"} Sep 05 06:52:09.507478 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:09.507438 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl" Sep 05 06:52:09.555527 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:09.555471 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5" podStartSLOduration=1.555452796 podStartE2EDuration="1.555452796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:52:09.530862876 +0000 UTC m=+1553.915406002" watchObservedRunningTime="2026-09-05 06:52:09.555452796 +0000 UTC m=+1553.939995922" Sep 05 06:52:09.556998 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:09.556966 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl" podStartSLOduration=1.556954822 podStartE2EDuration="1.556954822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:52:09.554188578 +0000 UTC m=+1553.938731701" watchObservedRunningTime="2026-09-05 06:52:09.556954822 +0000 UTC m=+1553.941497946" Sep 05 06:52:16.386118 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:16.386080 2565 scope.go:117] "RemoveContainer" containerID="db477351ad12de7cab7fdbcc6bfb956818da80b11ccacd473823c502c1a96454" Sep 05 06:52:20.516633 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:20.516602 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5" Sep 05 06:52:20.517070 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:20.516843 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl" Sep 05 06:52:36.814595 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:52:36.814563 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl_0b2d651a-289f-4daf-918c-3a58067ebb70/payload-processing/0.log" Sep 05 06:53:11.821391 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:11.821303 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl_0b2d651a-289f-4daf-918c-3a58067ebb70/payload-processing/0.log" Sep 05 06:53:16.427463 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:16.427433 2565 scope.go:117] "RemoveContainer" containerID="70fc5faee1a5a65aaff44135f43626ee5583e9fded36f7b2aa748f5c9b2432f9" Sep 05 06:53:17.079515 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:17.079473 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74"] Sep 05 06:53:17.084407 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:17.084385 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74" Sep 05 06:53:17.086718 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:17.086695 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-9db0f4\"" Sep 05 06:53:17.086838 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:17.086781 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-9db0f4-dockercfg-2wljw\"" Sep 05 06:53:17.097556 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:17.097533 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74"] Sep 05 06:53:17.254929 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:17.254893 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgxqd\" (UniqueName: \"kubernetes.io/projected/6e207a00-58d5-4cdc-aedc-4b114ede7174-kube-api-access-hgxqd\") pod \"payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74\" (UID: \"6e207a00-58d5-4cdc-aedc-4b114ede7174\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74" Sep 05 06:53:17.255117 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:17.255068 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e207a00-58d5-4cdc-aedc-4b114ede7174-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74\" (UID: \"6e207a00-58d5-4cdc-aedc-4b114ede7174\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74" Sep 05 06:53:17.356288 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:17.356243 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e207a00-58d5-4cdc-aedc-4b114ede7174-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74\" (UID: \"6e207a00-58d5-4cdc-aedc-4b114ede7174\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74" Sep 05 06:53:17.356507 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:17.356371 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgxqd\" (UniqueName: \"kubernetes.io/projected/6e207a00-58d5-4cdc-aedc-4b114ede7174-kube-api-access-hgxqd\") pod \"payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74\" (UID: \"6e207a00-58d5-4cdc-aedc-4b114ede7174\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74" Sep 05 06:53:17.356984 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:17.356958 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e207a00-58d5-4cdc-aedc-4b114ede7174-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74\" (UID: \"6e207a00-58d5-4cdc-aedc-4b114ede7174\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74" Sep 05 06:53:17.366601 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:17.366575 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgxqd\" (UniqueName: \"kubernetes.io/projected/6e207a00-58d5-4cdc-aedc-4b114ede7174-kube-api-access-hgxqd\") pod \"payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74\" (UID: \"6e207a00-58d5-4cdc-aedc-4b114ede7174\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74" Sep 05 06:53:17.394574 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:17.394546 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74" Sep 05 06:53:17.666443 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:17.666416 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74"] Sep 05 06:53:17.668035 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:53:17.668007 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e207a00_58d5_4cdc_aedc_4b114ede7174.slice/crio-dd9fb58595840bd95e8798727fffdabc1c045b6d55e6cb48272a248754bef474 WatchSource:0}: Error finding container dd9fb58595840bd95e8798727fffdabc1c045b6d55e6cb48272a248754bef474: Status 404 returned error can't find the container with id dd9fb58595840bd95e8798727fffdabc1c045b6d55e6cb48272a248754bef474 Sep 05 06:53:17.879874 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:17.879837 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74" event={"ID":"6e207a00-58d5-4cdc-aedc-4b114ede7174","Type":"ContainerStarted","Data":"54def5422c1bceae280cdcc24da7c92d952702ec12e69861eb352873cc46962a"} Sep 05 06:53:17.880053 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:17.879924 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74" event={"ID":"6e207a00-58d5-4cdc-aedc-4b114ede7174","Type":"ContainerStarted","Data":"dd9fb58595840bd95e8798727fffdabc1c045b6d55e6cb48272a248754bef474"} Sep 05 06:53:17.880144 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:17.880056 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74" Sep 05 06:53:17.896680 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:17.896631 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74" podStartSLOduration=0.896618235 podStartE2EDuration="896.618235ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:53:17.895801606 +0000 UTC m=+1622.280344731" watchObservedRunningTime="2026-09-05 06:53:17.896618235 +0000 UTC m=+1622.281161356" Sep 05 06:53:28.886617 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:28.886587 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74" Sep 05 06:53:47.147954 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:47.147921 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74"] Sep 05 06:53:47.148547 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:47.148162 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74" podUID="6e207a00-58d5-4cdc-aedc-4b114ede7174" containerName="payload-processing" containerID="cri-o://54def5422c1bceae280cdcc24da7c92d952702ec12e69861eb352873cc46962a" gracePeriod=30 Sep 05 06:53:47.396986 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:47.396961 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74" Sep 05 06:53:47.447706 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:47.447621 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgxqd\" (UniqueName: \"kubernetes.io/projected/6e207a00-58d5-4cdc-aedc-4b114ede7174-kube-api-access-hgxqd\") pod \"6e207a00-58d5-4cdc-aedc-4b114ede7174\" (UID: \"6e207a00-58d5-4cdc-aedc-4b114ede7174\") " Sep 05 06:53:47.447706 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:47.447663 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e207a00-58d5-4cdc-aedc-4b114ede7174-plugins-config-volume\") pod \"6e207a00-58d5-4cdc-aedc-4b114ede7174\" (UID: \"6e207a00-58d5-4cdc-aedc-4b114ede7174\") " Sep 05 06:53:47.448054 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:47.448025 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e207a00-58d5-4cdc-aedc-4b114ede7174-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e207a00-58d5-4cdc-aedc-4b114ede7174" (UID: "6e207a00-58d5-4cdc-aedc-4b114ede7174"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:53:47.450113 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:47.450080 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e207a00-58d5-4cdc-aedc-4b114ede7174-kube-api-access-hgxqd" (OuterVolumeSpecName: "kube-api-access-hgxqd") pod "6e207a00-58d5-4cdc-aedc-4b114ede7174" (UID: "6e207a00-58d5-4cdc-aedc-4b114ede7174"). InnerVolumeSpecName "kube-api-access-hgxqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:53:47.549291 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:47.549241 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgxqd\" (UniqueName: \"kubernetes.io/projected/6e207a00-58d5-4cdc-aedc-4b114ede7174-kube-api-access-hgxqd\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:53:47.549291 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:47.549278 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e207a00-58d5-4cdc-aedc-4b114ede7174-plugins-config-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:53:48.044535 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:48.044494 2565 generic.go:356] "Generic (PLEG): container finished" podID="6e207a00-58d5-4cdc-aedc-4b114ede7174" containerID="54def5422c1bceae280cdcc24da7c92d952702ec12e69861eb352873cc46962a" exitCode=0 Sep 05 06:53:48.044715 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:48.044544 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74" Sep 05 06:53:48.044715 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:48.044585 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74" event={"ID":"6e207a00-58d5-4cdc-aedc-4b114ede7174","Type":"ContainerDied","Data":"54def5422c1bceae280cdcc24da7c92d952702ec12e69861eb352873cc46962a"} Sep 05 06:53:48.044715 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:48.044642 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74" event={"ID":"6e207a00-58d5-4cdc-aedc-4b114ede7174","Type":"ContainerDied","Data":"dd9fb58595840bd95e8798727fffdabc1c045b6d55e6cb48272a248754bef474"} Sep 05 06:53:48.044715 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:48.044667 2565 scope.go:117] "RemoveContainer" containerID="54def5422c1bceae280cdcc24da7c92d952702ec12e69861eb352873cc46962a" Sep 05 06:53:48.054169 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:48.054154 2565 scope.go:117] "RemoveContainer" containerID="54def5422c1bceae280cdcc24da7c92d952702ec12e69861eb352873cc46962a" Sep 05 06:53:48.054462 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:53:48.054442 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54def5422c1bceae280cdcc24da7c92d952702ec12e69861eb352873cc46962a\": container with ID starting with 54def5422c1bceae280cdcc24da7c92d952702ec12e69861eb352873cc46962a not found: ID does not exist" containerID="54def5422c1bceae280cdcc24da7c92d952702ec12e69861eb352873cc46962a" Sep 05 06:53:48.054532 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:48.054470 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54def5422c1bceae280cdcc24da7c92d952702ec12e69861eb352873cc46962a"} err="failed to get container status \"54def5422c1bceae280cdcc24da7c92d952702ec12e69861eb352873cc46962a\": rpc error: code = NotFound desc = could not find container \"54def5422c1bceae280cdcc24da7c92d952702ec12e69861eb352873cc46962a\": container with ID starting with 54def5422c1bceae280cdcc24da7c92d952702ec12e69861eb352873cc46962a not found: ID does not exist" Sep 05 06:53:48.067065 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:48.067037 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74"] Sep 05 06:53:48.071496 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:48.071476 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-9db0f4-db9558477-x4q74"] Sep 05 06:53:48.167628 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:48.167594 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e207a00-58d5-4cdc-aedc-4b114ede7174" path="/var/lib/kubelet/pods/6e207a00-58d5-4cdc-aedc-4b114ede7174/volumes" Sep 05 06:53:58.123629 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:58.123592 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d36c73-xh9zr"] Sep 05 06:53:58.124042 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:58.124034 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e207a00-58d5-4cdc-aedc-4b114ede7174" containerName="payload-processing" Sep 05 06:53:58.124096 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:58.124046 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e207a00-58d5-4cdc-aedc-4b114ede7174" containerName="payload-processing" Sep 05 06:53:58.124134 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:58.124118 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e207a00-58d5-4cdc-aedc-4b114ede7174" containerName="payload-processing" Sep 05 06:53:58.128576 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:58.128559 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d36c73-xh9zr" Sep 05 06:53:58.131006 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:58.130978 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 05 06:53:58.131727 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:58.131695 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slfqj\"" Sep 05 06:53:58.134861 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:58.134831 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d36c73-xh9zr"] Sep 05 06:53:58.260369 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:58.260278 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b55fefe-758a-4025-b02d-f78fedceee10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d36c73-xh9zr\" (UID: \"9b55fefe-758a-4025-b02d-f78fedceee10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d36c73-xh9zr" Sep 05 06:53:58.361542 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:58.361505 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b55fefe-758a-4025-b02d-f78fedceee10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d36c73-xh9zr\" (UID: \"9b55fefe-758a-4025-b02d-f78fedceee10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d36c73-xh9zr" Sep 05 06:53:58.362096 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:58.362076 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b55fefe-758a-4025-b02d-f78fedceee10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d36c73-xh9zr\" (UID: \"9b55fefe-758a-4025-b02d-f78fedceee10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d36c73-xh9zr" Sep 05 06:53:58.441757 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:58.441662 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d36c73-xh9zr" Sep 05 06:53:58.714555 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:58.714400 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d36c73-xh9zr"] Sep 05 06:53:58.717077 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:53:58.717041 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b55fefe_758a_4025_b02d_f78fedceee10.slice/crio-8b33938dc240b65ad507cadd356c8ba692e9e50bf1c52ad002da7917520e55cc WatchSource:0}: Error finding container 8b33938dc240b65ad507cadd356c8ba692e9e50bf1c52ad002da7917520e55cc: Status 404 returned error can't find the container with id 8b33938dc240b65ad507cadd356c8ba692e9e50bf1c52ad002da7917520e55cc Sep 05 06:53:59.101668 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:59.101631 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d36c73-xh9zr" event={"ID":"9b55fefe-758a-4025-b02d-f78fedceee10","Type":"ContainerStarted","Data":"864e1da46a646a487d1a2a6333594b6a8abdd1eb61ddbca94ee449c701e5e111"} Sep 05 06:53:59.101846 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:59.101690 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d36c73-xh9zr" event={"ID":"9b55fefe-758a-4025-b02d-f78fedceee10","Type":"ContainerStarted","Data":"8b33938dc240b65ad507cadd356c8ba692e9e50bf1c52ad002da7917520e55cc"} Sep 05 06:53:59.118279 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:53:59.118205 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d36c73-xh9zr" podStartSLOduration=1.118183929 podStartE2EDuration="1.118183929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 06:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 06:53:59.115978871 +0000 UTC m=+1663.500522007" watchObservedRunningTime="2026-09-05 06:53:59.118183929 +0000 UTC m=+1663.502727054" Sep 05 06:54:00.108739 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:00.108708 2565 generic.go:356] "Generic (PLEG): container finished" podID="9b55fefe-758a-4025-b02d-f78fedceee10" containerID="864e1da46a646a487d1a2a6333594b6a8abdd1eb61ddbca94ee449c701e5e111" exitCode=0 Sep 05 06:54:00.109124 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:00.108757 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d36c73-xh9zr" event={"ID":"9b55fefe-758a-4025-b02d-f78fedceee10","Type":"ContainerDied","Data":"864e1da46a646a487d1a2a6333594b6a8abdd1eb61ddbca94ee449c701e5e111"} Sep 05 06:54:01.255068 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:01.255041 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d36c73-xh9zr" Sep 05 06:54:01.394848 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:01.394746 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b55fefe-758a-4025-b02d-f78fedceee10-maas-api-service-ca\") pod \"9b55fefe-758a-4025-b02d-f78fedceee10\" (UID: \"9b55fefe-758a-4025-b02d-f78fedceee10\") " Sep 05 06:54:01.395381 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:01.395350 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b55fefe-758a-4025-b02d-f78fedceee10-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9b55fefe-758a-4025-b02d-f78fedceee10" (UID: "9b55fefe-758a-4025-b02d-f78fedceee10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:54:01.395929 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:01.395898 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b55fefe-758a-4025-b02d-f78fedceee10-maas-api-service-ca\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:54:02.121352 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:02.121298 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d36c73-xh9zr" event={"ID":"9b55fefe-758a-4025-b02d-f78fedceee10","Type":"ContainerDied","Data":"8b33938dc240b65ad507cadd356c8ba692e9e50bf1c52ad002da7917520e55cc"} Sep 05 06:54:02.121526 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:02.121364 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d36c73-xh9zr" Sep 05 06:54:02.121526 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:02.121392 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b33938dc240b65ad507cadd356c8ba692e9e50bf1c52ad002da7917520e55cc" Sep 05 06:54:14.206041 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:14.206005 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd"] Sep 05 06:54:14.206517 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:14.206274 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd" podUID="39c3e90f-d3a3-450b-8476-90eed1d927ed" containerName="payload-pre-processing" containerID="cri-o://f5c87a86feb73ea22e768ba5678a0a27b398a02db6dc5ef622b1575b27308a6f" gracePeriod=30 Sep 05 06:54:14.451401 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:14.451366 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d36c73-xh9zr"] Sep 05 06:54:14.453378 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:14.453357 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d36c73-xh9zr"] Sep 05 06:54:14.473740 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:14.473679 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd" Sep 05 06:54:14.513000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:14.512962 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlz8r\" (UniqueName: \"kubernetes.io/projected/39c3e90f-d3a3-450b-8476-90eed1d927ed-kube-api-access-tlz8r\") pod \"39c3e90f-d3a3-450b-8476-90eed1d927ed\" (UID: \"39c3e90f-d3a3-450b-8476-90eed1d927ed\") " Sep 05 06:54:14.513166 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:14.513124 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39c3e90f-d3a3-450b-8476-90eed1d927ed-plugins-config-volume\") pod \"39c3e90f-d3a3-450b-8476-90eed1d927ed\" (UID: \"39c3e90f-d3a3-450b-8476-90eed1d927ed\") " Sep 05 06:54:14.513503 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:14.513477 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c3e90f-d3a3-450b-8476-90eed1d927ed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "39c3e90f-d3a3-450b-8476-90eed1d927ed" (UID: "39c3e90f-d3a3-450b-8476-90eed1d927ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:54:14.515199 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:14.515173 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c3e90f-d3a3-450b-8476-90eed1d927ed-kube-api-access-tlz8r" (OuterVolumeSpecName: "kube-api-access-tlz8r") pod "39c3e90f-d3a3-450b-8476-90eed1d927ed" (UID: "39c3e90f-d3a3-450b-8476-90eed1d927ed"). InnerVolumeSpecName "kube-api-access-tlz8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:54:14.614251 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:14.614213 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39c3e90f-d3a3-450b-8476-90eed1d927ed-plugins-config-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:54:14.614251 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:14.614244 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlz8r\" (UniqueName: \"kubernetes.io/projected/39c3e90f-d3a3-450b-8476-90eed1d927ed-kube-api-access-tlz8r\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:54:15.192129 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:15.192100 2565 generic.go:356] "Generic (PLEG): container finished" podID="39c3e90f-d3a3-450b-8476-90eed1d927ed" containerID="f5c87a86feb73ea22e768ba5678a0a27b398a02db6dc5ef622b1575b27308a6f" exitCode=0 Sep 05 06:54:15.192387 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:15.192141 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd" Sep 05 06:54:15.192387 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:15.192189 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd" event={"ID":"39c3e90f-d3a3-450b-8476-90eed1d927ed","Type":"ContainerDied","Data":"f5c87a86feb73ea22e768ba5678a0a27b398a02db6dc5ef622b1575b27308a6f"} Sep 05 06:54:15.192387 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:15.192246 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd" event={"ID":"39c3e90f-d3a3-450b-8476-90eed1d927ed","Type":"ContainerDied","Data":"46905ddd803a022d46dc6aab85f4053062f699932226f37a9936f8b81db470de"} Sep 05 06:54:15.192387 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:15.192276 2565 scope.go:117] "RemoveContainer" containerID="f5c87a86feb73ea22e768ba5678a0a27b398a02db6dc5ef622b1575b27308a6f" Sep 05 06:54:15.201703 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:15.201685 2565 scope.go:117] "RemoveContainer" containerID="f5c87a86feb73ea22e768ba5678a0a27b398a02db6dc5ef622b1575b27308a6f" Sep 05 06:54:15.201995 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:54:15.201976 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c87a86feb73ea22e768ba5678a0a27b398a02db6dc5ef622b1575b27308a6f\": container with ID starting with f5c87a86feb73ea22e768ba5678a0a27b398a02db6dc5ef622b1575b27308a6f not found: ID does not exist" containerID="f5c87a86feb73ea22e768ba5678a0a27b398a02db6dc5ef622b1575b27308a6f" Sep 05 06:54:15.202056 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:15.202008 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c87a86feb73ea22e768ba5678a0a27b398a02db6dc5ef622b1575b27308a6f"} err="failed to get container status \"f5c87a86feb73ea22e768ba5678a0a27b398a02db6dc5ef622b1575b27308a6f\": rpc error: code = NotFound desc = could not find container \"f5c87a86feb73ea22e768ba5678a0a27b398a02db6dc5ef622b1575b27308a6f\": container with ID starting with f5c87a86feb73ea22e768ba5678a0a27b398a02db6dc5ef622b1575b27308a6f not found: ID does not exist" Sep 05 06:54:15.214662 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:15.214637 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd"] Sep 05 06:54:15.217648 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:15.217626 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d36c73-586d4f8fbb-wnhgd"] Sep 05 06:54:16.167048 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:16.167015 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39c3e90f-d3a3-450b-8476-90eed1d927ed" path="/var/lib/kubelet/pods/39c3e90f-d3a3-450b-8476-90eed1d927ed/volumes" Sep 05 06:54:16.167492 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:54:16.167472 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b55fefe-758a-4025-b02d-f78fedceee10" path="/var/lib/kubelet/pods/9b55fefe-758a-4025-b02d-f78fedceee10/volumes" Sep 05 06:55:03.460501 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:03.460465 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx"] Sep 05 06:55:03.461022 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:03.460681 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" podUID="d36796a4-e4e2-4c53-83e6-0e7b56f3d854" containerName="istio-proxy" containerID="cri-o://123904d04e299aa8cce374bdf1857b16bb964fc52ed201d6caa1a38cbc87c8e0" gracePeriod=30 Sep 05 06:55:03.492548 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:03.492512 2565 patch_prober.go:28] interesting pod/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 05 06:55:03.492715 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:03.492570 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" podUID="d36796a4-e4e2-4c53-83e6-0e7b56f3d854" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 05 06:55:08.719458 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.719431 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:55:08.820599 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.820518 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istio-envoy\") pod \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " Sep 05 06:55:08.820599 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.820578 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istiod-ca-cert\") pod \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " Sep 05 06:55:08.820599 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.820599 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-workload-socket\") pod \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " Sep 05 06:55:08.820872 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.820639 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-workload-certs\") pod \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " Sep 05 06:55:08.820872 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.820657 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-th4sz\" (UniqueName: \"kubernetes.io/projected/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-kube-api-access-th4sz\") pod \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " Sep 05 06:55:08.820872 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.820762 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istio-token\") pod \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " Sep 05 06:55:08.820872 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.820788 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-credential-socket\") pod \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " Sep 05 06:55:08.820872 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.820816 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istio-data\") pod \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " Sep 05 06:55:08.820872 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.820838 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istio-podinfo\") pod \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\" (UID: \"d36796a4-e4e2-4c53-83e6-0e7b56f3d854\") " Sep 05 06:55:08.821170 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.820902 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d36796a4-e4e2-4c53-83e6-0e7b56f3d854" (UID: "d36796a4-e4e2-4c53-83e6-0e7b56f3d854"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:55:08.821170 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.820954 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d36796a4-e4e2-4c53-83e6-0e7b56f3d854" (UID: "d36796a4-e4e2-4c53-83e6-0e7b56f3d854"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:55:08.821170 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.821045 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d36796a4-e4e2-4c53-83e6-0e7b56f3d854" (UID: "d36796a4-e4e2-4c53-83e6-0e7b56f3d854"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:55:08.821347 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.821184 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istio-data" (OuterVolumeSpecName: "istio-data") pod "d36796a4-e4e2-4c53-83e6-0e7b56f3d854" (UID: "d36796a4-e4e2-4c53-83e6-0e7b56f3d854"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:55:08.821347 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.821232 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d36796a4-e4e2-4c53-83e6-0e7b56f3d854" (UID: "d36796a4-e4e2-4c53-83e6-0e7b56f3d854"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:55:08.821347 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.821284 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-workload-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:08.821506 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.821444 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-workload-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:08.821546 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.821511 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-credential-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:08.821546 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.821524 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istio-data\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:08.823000 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.822966 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-kube-api-access-th4sz" (OuterVolumeSpecName: "kube-api-access-th4sz") pod "d36796a4-e4e2-4c53-83e6-0e7b56f3d854" (UID: "d36796a4-e4e2-4c53-83e6-0e7b56f3d854"). InnerVolumeSpecName "kube-api-access-th4sz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:55:08.823120 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.823041 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d36796a4-e4e2-4c53-83e6-0e7b56f3d854" (UID: "d36796a4-e4e2-4c53-83e6-0e7b56f3d854"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:55:08.823601 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.823579 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istio-token" (OuterVolumeSpecName: "istio-token") pod "d36796a4-e4e2-4c53-83e6-0e7b56f3d854" (UID: "d36796a4-e4e2-4c53-83e6-0e7b56f3d854"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:55:08.823681 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.823634 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d36796a4-e4e2-4c53-83e6-0e7b56f3d854" (UID: "d36796a4-e4e2-4c53-83e6-0e7b56f3d854"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 05 06:55:08.922227 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.922194 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istio-token\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:08.922227 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.922224 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istio-podinfo\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:08.922227 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.922233 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istio-envoy\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:08.922495 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.922240 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-istiod-ca-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:08.922495 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:08.922249 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th4sz\" (UniqueName: \"kubernetes.io/projected/d36796a4-e4e2-4c53-83e6-0e7b56f3d854-kube-api-access-th4sz\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:09.477914 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:09.477883 2565 generic.go:356] "Generic (PLEG): container finished" podID="d36796a4-e4e2-4c53-83e6-0e7b56f3d854" containerID="123904d04e299aa8cce374bdf1857b16bb964fc52ed201d6caa1a38cbc87c8e0" exitCode=0 Sep 05 06:55:09.478097 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:09.477932 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" Sep 05 06:55:09.478097 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:09.477970 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" event={"ID":"d36796a4-e4e2-4c53-83e6-0e7b56f3d854","Type":"ContainerDied","Data":"123904d04e299aa8cce374bdf1857b16bb964fc52ed201d6caa1a38cbc87c8e0"} Sep 05 06:55:09.478097 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:09.478032 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx" event={"ID":"d36796a4-e4e2-4c53-83e6-0e7b56f3d854","Type":"ContainerDied","Data":"9dc2a9ce7666b37607116e7042eb52c8e3d339306f965b635acdef49c113f360"} Sep 05 06:55:09.478097 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:09.478054 2565 scope.go:117] "RemoveContainer" containerID="123904d04e299aa8cce374bdf1857b16bb964fc52ed201d6caa1a38cbc87c8e0" Sep 05 06:55:09.488128 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:09.488112 2565 scope.go:117] "RemoveContainer" containerID="1396c5e65f0c2bb79fa844a754a2797439a37f9ec26bb61525de25ba4758f007" Sep 05 06:55:09.496583 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:09.496558 2565 scope.go:117] "RemoveContainer" containerID="123904d04e299aa8cce374bdf1857b16bb964fc52ed201d6caa1a38cbc87c8e0" Sep 05 06:55:09.496853 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:55:09.496834 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"123904d04e299aa8cce374bdf1857b16bb964fc52ed201d6caa1a38cbc87c8e0\": container with ID starting with 123904d04e299aa8cce374bdf1857b16bb964fc52ed201d6caa1a38cbc87c8e0 not found: ID does not exist" containerID="123904d04e299aa8cce374bdf1857b16bb964fc52ed201d6caa1a38cbc87c8e0" Sep 05 06:55:09.496919 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:09.496861 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"123904d04e299aa8cce374bdf1857b16bb964fc52ed201d6caa1a38cbc87c8e0"} err="failed to get container status \"123904d04e299aa8cce374bdf1857b16bb964fc52ed201d6caa1a38cbc87c8e0\": rpc error: code = NotFound desc = could not find container \"123904d04e299aa8cce374bdf1857b16bb964fc52ed201d6caa1a38cbc87c8e0\": container with ID starting with 123904d04e299aa8cce374bdf1857b16bb964fc52ed201d6caa1a38cbc87c8e0 not found: ID does not exist" Sep 05 06:55:09.496919 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:09.496876 2565 scope.go:117] "RemoveContainer" containerID="1396c5e65f0c2bb79fa844a754a2797439a37f9ec26bb61525de25ba4758f007" Sep 05 06:55:09.497111 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:55:09.497094 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1396c5e65f0c2bb79fa844a754a2797439a37f9ec26bb61525de25ba4758f007\": container with ID starting with 1396c5e65f0c2bb79fa844a754a2797439a37f9ec26bb61525de25ba4758f007 not found: ID does not exist" containerID="1396c5e65f0c2bb79fa844a754a2797439a37f9ec26bb61525de25ba4758f007" Sep 05 06:55:09.497152 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:09.497116 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1396c5e65f0c2bb79fa844a754a2797439a37f9ec26bb61525de25ba4758f007"} err="failed to get container status \"1396c5e65f0c2bb79fa844a754a2797439a37f9ec26bb61525de25ba4758f007\": rpc error: code = NotFound desc = could not find container \"1396c5e65f0c2bb79fa844a754a2797439a37f9ec26bb61525de25ba4758f007\": container with ID starting with 1396c5e65f0c2bb79fa844a754a2797439a37f9ec26bb61525de25ba4758f007 not found: ID does not exist" Sep 05 06:55:09.501259 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:09.501237 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx"] Sep 05 06:55:09.504991 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:09.504971 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-d36c73-openshift-default-5865c85dc8-fv4lx"] Sep 05 06:55:10.166463 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:10.166430 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d36796a4-e4e2-4c53-83e6-0e7b56f3d854" path="/var/lib/kubelet/pods/d36796a4-e4e2-4c53-83e6-0e7b56f3d854/volumes" Sep 05 06:55:14.556428 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:14.556389 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl"] Sep 05 06:55:14.556917 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:14.556615 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl" podUID="0b2d651a-289f-4daf-918c-3a58067ebb70" containerName="payload-processing" containerID="cri-o://179fd9400b565781ec3d8c37dd2275123264643dd1c1727cf4d82ab2e4d49286" gracePeriod=30 Sep 05 06:55:14.563638 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:14.563602 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5"] Sep 05 06:55:14.564334 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:14.563992 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5" podUID="13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60" containerName="payload-pre-processing" containerID="cri-o://edbade95fc902b9c1998ea28107a50b91c142f3182486f0a5cd93c052690c2d8" gracePeriod=30 Sep 05 06:55:14.826327 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:14.826288 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl" Sep 05 06:55:14.850702 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:14.850674 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5" Sep 05 06:55:14.981425 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:14.981393 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jcvs\" (UniqueName: \"kubernetes.io/projected/0b2d651a-289f-4daf-918c-3a58067ebb70-kube-api-access-9jcvs\") pod \"0b2d651a-289f-4daf-918c-3a58067ebb70\" (UID: \"0b2d651a-289f-4daf-918c-3a58067ebb70\") " Sep 05 06:55:14.981425 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:14.981439 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gr52\" (UniqueName: \"kubernetes.io/projected/13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60-kube-api-access-6gr52\") pod \"13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60\" (UID: \"13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60\") " Sep 05 06:55:14.981657 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:14.981526 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b2d651a-289f-4daf-918c-3a58067ebb70-plugins-config-volume\") pod \"0b2d651a-289f-4daf-918c-3a58067ebb70\" (UID: \"0b2d651a-289f-4daf-918c-3a58067ebb70\") " Sep 05 06:55:14.981657 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:14.981557 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60-plugins-config-volume\") pod \"13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60\" (UID: \"13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60\") " Sep 05 06:55:14.981918 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:14.981894 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b2d651a-289f-4daf-918c-3a58067ebb70-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0b2d651a-289f-4daf-918c-3a58067ebb70" (UID: "0b2d651a-289f-4daf-918c-3a58067ebb70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:55:14.982008 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:14.981976 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60" (UID: "13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:55:14.983810 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:14.983790 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60-kube-api-access-6gr52" (OuterVolumeSpecName: "kube-api-access-6gr52") pod "13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60" (UID: "13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60"). InnerVolumeSpecName "kube-api-access-6gr52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:55:14.983875 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:14.983832 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b2d651a-289f-4daf-918c-3a58067ebb70-kube-api-access-9jcvs" (OuterVolumeSpecName: "kube-api-access-9jcvs") pod "0b2d651a-289f-4daf-918c-3a58067ebb70" (UID: "0b2d651a-289f-4daf-918c-3a58067ebb70"). InnerVolumeSpecName "kube-api-access-9jcvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:55:15.082562 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:15.082481 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jcvs\" (UniqueName: \"kubernetes.io/projected/0b2d651a-289f-4daf-918c-3a58067ebb70-kube-api-access-9jcvs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:15.082562 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:15.082506 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gr52\" (UniqueName: \"kubernetes.io/projected/13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60-kube-api-access-6gr52\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:15.082562 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:15.082515 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b2d651a-289f-4daf-918c-3a58067ebb70-plugins-config-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:15.082562 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:15.082524 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60-plugins-config-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:15.525831 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:15.525792 2565 generic.go:356] "Generic (PLEG): container finished" podID="13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60" containerID="edbade95fc902b9c1998ea28107a50b91c142f3182486f0a5cd93c052690c2d8" exitCode=0 Sep 05 06:55:15.526039 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:15.525849 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5" Sep 05 06:55:15.526039 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:15.525890 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5" event={"ID":"13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60","Type":"ContainerDied","Data":"edbade95fc902b9c1998ea28107a50b91c142f3182486f0a5cd93c052690c2d8"} Sep 05 06:55:15.526039 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:15.525967 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5" event={"ID":"13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60","Type":"ContainerDied","Data":"3cd1a373814539426ed2dd6292e1ced8a9c874f362e1406d27a280f8fbd55cd1"} Sep 05 06:55:15.526039 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:15.525995 2565 scope.go:117] "RemoveContainer" containerID="edbade95fc902b9c1998ea28107a50b91c142f3182486f0a5cd93c052690c2d8" Sep 05 06:55:15.527265 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:15.527243 2565 generic.go:356] "Generic (PLEG): container finished" podID="0b2d651a-289f-4daf-918c-3a58067ebb70" containerID="179fd9400b565781ec3d8c37dd2275123264643dd1c1727cf4d82ab2e4d49286" exitCode=0 Sep 05 06:55:15.527463 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:15.527324 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl" Sep 05 06:55:15.527463 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:15.527341 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl" event={"ID":"0b2d651a-289f-4daf-918c-3a58067ebb70","Type":"ContainerDied","Data":"179fd9400b565781ec3d8c37dd2275123264643dd1c1727cf4d82ab2e4d49286"} Sep 05 06:55:15.527463 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:15.527393 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl" event={"ID":"0b2d651a-289f-4daf-918c-3a58067ebb70","Type":"ContainerDied","Data":"c151a60415cf74c622a59b02e99fbc718badec1a4293d9ade71e1b19a80bc153"} Sep 05 06:55:15.536846 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:15.536682 2565 scope.go:117] "RemoveContainer" containerID="edbade95fc902b9c1998ea28107a50b91c142f3182486f0a5cd93c052690c2d8" Sep 05 06:55:15.536993 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:55:15.536968 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edbade95fc902b9c1998ea28107a50b91c142f3182486f0a5cd93c052690c2d8\": container with ID starting with edbade95fc902b9c1998ea28107a50b91c142f3182486f0a5cd93c052690c2d8 not found: ID does not exist" containerID="edbade95fc902b9c1998ea28107a50b91c142f3182486f0a5cd93c052690c2d8" Sep 05 06:55:15.537059 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:15.537004 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edbade95fc902b9c1998ea28107a50b91c142f3182486f0a5cd93c052690c2d8"} err="failed to get container status \"edbade95fc902b9c1998ea28107a50b91c142f3182486f0a5cd93c052690c2d8\": rpc error: code = NotFound desc = could not find container \"edbade95fc902b9c1998ea28107a50b91c142f3182486f0a5cd93c052690c2d8\": container with ID starting with edbade95fc902b9c1998ea28107a50b91c142f3182486f0a5cd93c052690c2d8 not found: ID does not exist" Sep 05 06:55:15.537059 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:15.537028 2565 scope.go:117] "RemoveContainer" containerID="179fd9400b565781ec3d8c37dd2275123264643dd1c1727cf4d82ab2e4d49286" Sep 05 06:55:15.548407 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:15.548384 2565 scope.go:117] "RemoveContainer" containerID="179fd9400b565781ec3d8c37dd2275123264643dd1c1727cf4d82ab2e4d49286" Sep 05 06:55:15.548716 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:55:15.548691 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"179fd9400b565781ec3d8c37dd2275123264643dd1c1727cf4d82ab2e4d49286\": container with ID starting with 179fd9400b565781ec3d8c37dd2275123264643dd1c1727cf4d82ab2e4d49286 not found: ID does not exist" containerID="179fd9400b565781ec3d8c37dd2275123264643dd1c1727cf4d82ab2e4d49286" Sep 05 06:55:15.548801 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:15.548726 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179fd9400b565781ec3d8c37dd2275123264643dd1c1727cf4d82ab2e4d49286"} err="failed to get container status \"179fd9400b565781ec3d8c37dd2275123264643dd1c1727cf4d82ab2e4d49286\": rpc error: code = NotFound desc = could not find container \"179fd9400b565781ec3d8c37dd2275123264643dd1c1727cf4d82ab2e4d49286\": container with ID starting with 179fd9400b565781ec3d8c37dd2275123264643dd1c1727cf4d82ab2e4d49286 not found: ID does not exist" Sep 05 06:55:15.549072 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:15.549058 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5"] Sep 05 06:55:15.553136 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:15.553117 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b49086-56576f558b-vfbp5"] Sep 05 06:55:15.564183 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:15.564163 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl"] Sep 05 06:55:15.570742 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:15.570721 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b49086-6fbf79ccbf-z5jdl"] Sep 05 06:55:16.166148 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:16.166114 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b2d651a-289f-4daf-918c-3a58067ebb70" path="/var/lib/kubelet/pods/0b2d651a-289f-4daf-918c-3a58067ebb70/volumes" Sep 05 06:55:16.166611 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:16.166590 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60" path="/var/lib/kubelet/pods/13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60/volumes" Sep 05 06:55:22.034354 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.034322 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-183ae6-vjwz9"] Sep 05 06:55:22.034732 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.034691 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39c3e90f-d3a3-450b-8476-90eed1d927ed" containerName="payload-pre-processing" Sep 05 06:55:22.034732 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.034702 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="39c3e90f-d3a3-450b-8476-90eed1d927ed" containerName="payload-pre-processing" Sep 05 06:55:22.034732 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.034714 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b2d651a-289f-4daf-918c-3a58067ebb70" containerName="payload-processing" Sep 05 06:55:22.034732 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.034720 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b2d651a-289f-4daf-918c-3a58067ebb70" containerName="payload-processing" Sep 05 06:55:22.034732 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.034731 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b55fefe-758a-4025-b02d-f78fedceee10" containerName="revoke-keys" Sep 05 06:55:22.034906 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.034737 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b55fefe-758a-4025-b02d-f78fedceee10" containerName="revoke-keys" Sep 05 06:55:22.034906 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.034746 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60" containerName="payload-pre-processing" Sep 05 06:55:22.034906 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.034751 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60" containerName="payload-pre-processing" Sep 05 06:55:22.034906 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.034762 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d36796a4-e4e2-4c53-83e6-0e7b56f3d854" containerName="istio-proxy" Sep 05 06:55:22.034906 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.034769 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d36796a4-e4e2-4c53-83e6-0e7b56f3d854" containerName="istio-proxy" Sep 05 06:55:22.034906 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.034777 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d36796a4-e4e2-4c53-83e6-0e7b56f3d854" containerName="istio-proxy" Sep 05 06:55:22.034906 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.034782 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d36796a4-e4e2-4c53-83e6-0e7b56f3d854" containerName="istio-proxy" Sep 05 06:55:22.034906 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.034838 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="13a9ec1f-f301-4cc9-9e8d-05cf09ee4e60" containerName="payload-pre-processing" Sep 05 06:55:22.034906 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.034845 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="39c3e90f-d3a3-450b-8476-90eed1d927ed" containerName="payload-pre-processing" Sep 05 06:55:22.034906 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.034851 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b55fefe-758a-4025-b02d-f78fedceee10" containerName="revoke-keys" Sep 05 06:55:22.034906 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.034857 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d36796a4-e4e2-4c53-83e6-0e7b56f3d854" containerName="istio-proxy" Sep 05 06:55:22.034906 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.034864 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d36796a4-e4e2-4c53-83e6-0e7b56f3d854" containerName="istio-proxy" Sep 05 06:55:22.034906 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.034869 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b2d651a-289f-4daf-918c-3a58067ebb70" containerName="payload-processing" Sep 05 06:55:22.039192 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.039176 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-183ae6-vjwz9" Sep 05 06:55:22.041435 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.041410 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slfqj\"" Sep 05 06:55:22.042209 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.042184 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 05 06:55:22.044639 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.044617 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-183ae6-vjwz9"] Sep 05 06:55:22.152144 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.152111 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8160622-0e2a-431a-9952-7bdf6c0cc373-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-183ae6-vjwz9\" (UID: \"d8160622-0e2a-431a-9952-7bdf6c0cc373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-183ae6-vjwz9" Sep 05 06:55:22.253611 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.253571 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8160622-0e2a-431a-9952-7bdf6c0cc373-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-183ae6-vjwz9\" (UID: \"d8160622-0e2a-431a-9952-7bdf6c0cc373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-183ae6-vjwz9" Sep 05 06:55:22.254203 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.254183 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8160622-0e2a-431a-9952-7bdf6c0cc373-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-183ae6-vjwz9\" (UID: \"d8160622-0e2a-431a-9952-7bdf6c0cc373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-183ae6-vjwz9" Sep 05 06:55:22.351838 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.351810 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-183ae6-vjwz9" Sep 05 06:55:22.622047 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:22.622024 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-183ae6-vjwz9"] Sep 05 06:55:22.624647 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:55:22.624614 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8160622_0e2a_431a_9952_7bdf6c0cc373.slice/crio-1312fa214da40925b6566cdd48ef565758a5b86e51926fb0610e85058315f118 WatchSource:0}: Error finding container 1312fa214da40925b6566cdd48ef565758a5b86e51926fb0610e85058315f118: Status 404 returned error can't find the container with id 1312fa214da40925b6566cdd48ef565758a5b86e51926fb0610e85058315f118 Sep 05 06:55:23.569008 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:23.568923 2565 generic.go:356] "Generic (PLEG): container finished" podID="d8160622-0e2a-431a-9952-7bdf6c0cc373" containerID="1a7212f7a1f62ad5b8dbdc72138eb9fcec02e1294bc5e04b90d09021a8264cb1" exitCode=0 Sep 05 06:55:23.569008 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:23.568996 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-183ae6-vjwz9" event={"ID":"d8160622-0e2a-431a-9952-7bdf6c0cc373","Type":"ContainerDied","Data":"1a7212f7a1f62ad5b8dbdc72138eb9fcec02e1294bc5e04b90d09021a8264cb1"} Sep 05 06:55:23.569440 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:23.569048 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-183ae6-vjwz9" event={"ID":"d8160622-0e2a-431a-9952-7bdf6c0cc373","Type":"ContainerStarted","Data":"1312fa214da40925b6566cdd48ef565758a5b86e51926fb0610e85058315f118"} Sep 05 06:55:24.706093 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:24.706066 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-183ae6-vjwz9" Sep 05 06:55:24.880083 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:24.880034 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8160622-0e2a-431a-9952-7bdf6c0cc373-maas-api-service-ca\") pod \"d8160622-0e2a-431a-9952-7bdf6c0cc373\" (UID: \"d8160622-0e2a-431a-9952-7bdf6c0cc373\") " Sep 05 06:55:24.880445 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:24.880420 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8160622-0e2a-431a-9952-7bdf6c0cc373-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d8160622-0e2a-431a-9952-7bdf6c0cc373" (UID: "d8160622-0e2a-431a-9952-7bdf6c0cc373"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:55:24.981466 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:24.981431 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8160622-0e2a-431a-9952-7bdf6c0cc373-maas-api-service-ca\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:25.580477 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:25.580447 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-183ae6-vjwz9" Sep 05 06:55:25.580673 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:25.580491 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-183ae6-vjwz9" event={"ID":"d8160622-0e2a-431a-9952-7bdf6c0cc373","Type":"ContainerDied","Data":"1312fa214da40925b6566cdd48ef565758a5b86e51926fb0610e85058315f118"} Sep 05 06:55:25.580673 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:25.580535 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1312fa214da40925b6566cdd48ef565758a5b86e51926fb0610e85058315f118" Sep 05 06:55:32.562128 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:32.562088 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-183ae6-29809845-zhmnw"] Sep 05 06:55:32.565091 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:32.565065 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-183ae6-29809845-zhmnw"] Sep 05 06:55:32.585256 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:32.585229 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd"] Sep 05 06:55:32.585548 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:32.585506 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd" podUID="b69b6519-cb9a-482a-8fbd-902283fe6a42" containerName="payload-processing" containerID="cri-o://dfdc139d06f8ad8012a5b58af70925d6d9361a563f61ba26f10205e5a99dac52" gracePeriod=30 Sep 05 06:55:32.593615 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:32.593591 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq"] Sep 05 06:55:32.594038 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:32.593820 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq" podUID="250ab2e7-8101-463f-8dee-591d425a1469" containerName="payload-pre-processing" containerID="cri-o://c572c31c3218eee82b74935c2f82d12209dcc12b983024d175c6e5d45f502302" gracePeriod=30 Sep 05 06:55:32.768093 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:32.768061 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-183ae6-vjwz9"] Sep 05 06:55:32.771474 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:32.771425 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-183ae6-vjwz9"] Sep 05 06:55:32.875657 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:32.875638 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd" Sep 05 06:55:32.878488 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:32.878471 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq" Sep 05 06:55:32.950176 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:32.950145 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fmkg\" (UniqueName: \"kubernetes.io/projected/250ab2e7-8101-463f-8dee-591d425a1469-kube-api-access-5fmkg\") pod \"250ab2e7-8101-463f-8dee-591d425a1469\" (UID: \"250ab2e7-8101-463f-8dee-591d425a1469\") " Sep 05 06:55:32.950376 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:32.950260 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250ab2e7-8101-463f-8dee-591d425a1469-plugins-config-volume\") pod \"250ab2e7-8101-463f-8dee-591d425a1469\" (UID: \"250ab2e7-8101-463f-8dee-591d425a1469\") " Sep 05 06:55:32.950376 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:32.950286 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b69b6519-cb9a-482a-8fbd-902283fe6a42-plugins-config-volume\") pod \"b69b6519-cb9a-482a-8fbd-902283fe6a42\" (UID: \"b69b6519-cb9a-482a-8fbd-902283fe6a42\") " Sep 05 06:55:32.950376 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:32.950318 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pb9bs\" (UniqueName: \"kubernetes.io/projected/b69b6519-cb9a-482a-8fbd-902283fe6a42-kube-api-access-pb9bs\") pod \"b69b6519-cb9a-482a-8fbd-902283fe6a42\" (UID: \"b69b6519-cb9a-482a-8fbd-902283fe6a42\") " Sep 05 06:55:32.950667 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:32.950646 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b69b6519-cb9a-482a-8fbd-902283fe6a42-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b69b6519-cb9a-482a-8fbd-902283fe6a42" (UID: "b69b6519-cb9a-482a-8fbd-902283fe6a42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:55:32.950667 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:32.950648 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250ab2e7-8101-463f-8dee-591d425a1469-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "250ab2e7-8101-463f-8dee-591d425a1469" (UID: "250ab2e7-8101-463f-8dee-591d425a1469"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:55:32.952542 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:32.952502 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/250ab2e7-8101-463f-8dee-591d425a1469-kube-api-access-5fmkg" (OuterVolumeSpecName: "kube-api-access-5fmkg") pod "250ab2e7-8101-463f-8dee-591d425a1469" (UID: "250ab2e7-8101-463f-8dee-591d425a1469"). InnerVolumeSpecName "kube-api-access-5fmkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:55:32.952653 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:32.952629 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b69b6519-cb9a-482a-8fbd-902283fe6a42-kube-api-access-pb9bs" (OuterVolumeSpecName: "kube-api-access-pb9bs") pod "b69b6519-cb9a-482a-8fbd-902283fe6a42" (UID: "b69b6519-cb9a-482a-8fbd-902283fe6a42"). InnerVolumeSpecName "kube-api-access-pb9bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:55:33.050948 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:33.050917 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb9bs\" (UniqueName: \"kubernetes.io/projected/b69b6519-cb9a-482a-8fbd-902283fe6a42-kube-api-access-pb9bs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:33.050948 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:33.050943 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fmkg\" (UniqueName: \"kubernetes.io/projected/250ab2e7-8101-463f-8dee-591d425a1469-kube-api-access-5fmkg\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:33.050948 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:33.050952 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250ab2e7-8101-463f-8dee-591d425a1469-plugins-config-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:33.051148 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:33.050962 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b69b6519-cb9a-482a-8fbd-902283fe6a42-plugins-config-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:33.625350 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:33.625320 2565 generic.go:356] "Generic (PLEG): container finished" podID="250ab2e7-8101-463f-8dee-591d425a1469" containerID="c572c31c3218eee82b74935c2f82d12209dcc12b983024d175c6e5d45f502302" exitCode=0 Sep 05 06:55:33.625812 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:33.625382 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq" Sep 05 06:55:33.625812 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:33.625407 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq" event={"ID":"250ab2e7-8101-463f-8dee-591d425a1469","Type":"ContainerDied","Data":"c572c31c3218eee82b74935c2f82d12209dcc12b983024d175c6e5d45f502302"} Sep 05 06:55:33.625812 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:33.625467 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq" event={"ID":"250ab2e7-8101-463f-8dee-591d425a1469","Type":"ContainerDied","Data":"5536e138c4858df133f4cc8d8e0d2c084f1789f919a96df8a3db6c4c2cd473f2"} Sep 05 06:55:33.625812 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:33.625505 2565 scope.go:117] "RemoveContainer" containerID="c572c31c3218eee82b74935c2f82d12209dcc12b983024d175c6e5d45f502302" Sep 05 06:55:33.626991 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:33.626826 2565 generic.go:356] "Generic (PLEG): container finished" podID="b69b6519-cb9a-482a-8fbd-902283fe6a42" containerID="dfdc139d06f8ad8012a5b58af70925d6d9361a563f61ba26f10205e5a99dac52" exitCode=0 Sep 05 06:55:33.626991 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:33.626886 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd" Sep 05 06:55:33.626991 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:33.626887 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd" event={"ID":"b69b6519-cb9a-482a-8fbd-902283fe6a42","Type":"ContainerDied","Data":"dfdc139d06f8ad8012a5b58af70925d6d9361a563f61ba26f10205e5a99dac52"} Sep 05 06:55:33.627174 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:33.626997 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd" event={"ID":"b69b6519-cb9a-482a-8fbd-902283fe6a42","Type":"ContainerDied","Data":"af964c2f79e6363e8b8a335e16fd9db398566f6770b41eaf81d38be1a3ee298e"} Sep 05 06:55:33.637163 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:33.637145 2565 scope.go:117] "RemoveContainer" containerID="c572c31c3218eee82b74935c2f82d12209dcc12b983024d175c6e5d45f502302" Sep 05 06:55:33.637511 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:55:33.637482 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c572c31c3218eee82b74935c2f82d12209dcc12b983024d175c6e5d45f502302\": container with ID starting with c572c31c3218eee82b74935c2f82d12209dcc12b983024d175c6e5d45f502302 not found: ID does not exist" containerID="c572c31c3218eee82b74935c2f82d12209dcc12b983024d175c6e5d45f502302" Sep 05 06:55:33.637624 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:33.637511 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c572c31c3218eee82b74935c2f82d12209dcc12b983024d175c6e5d45f502302"} err="failed to get container status \"c572c31c3218eee82b74935c2f82d12209dcc12b983024d175c6e5d45f502302\": rpc error: code = NotFound desc = could not find container \"c572c31c3218eee82b74935c2f82d12209dcc12b983024d175c6e5d45f502302\": container with ID starting with c572c31c3218eee82b74935c2f82d12209dcc12b983024d175c6e5d45f502302 not found: ID does not exist" Sep 05 06:55:33.637624 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:33.637535 2565 scope.go:117] "RemoveContainer" containerID="dfdc139d06f8ad8012a5b58af70925d6d9361a563f61ba26f10205e5a99dac52" Sep 05 06:55:33.645809 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:33.645787 2565 scope.go:117] "RemoveContainer" containerID="dfdc139d06f8ad8012a5b58af70925d6d9361a563f61ba26f10205e5a99dac52" Sep 05 06:55:33.646058 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:55:33.646041 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfdc139d06f8ad8012a5b58af70925d6d9361a563f61ba26f10205e5a99dac52\": container with ID starting with dfdc139d06f8ad8012a5b58af70925d6d9361a563f61ba26f10205e5a99dac52 not found: ID does not exist" containerID="dfdc139d06f8ad8012a5b58af70925d6d9361a563f61ba26f10205e5a99dac52" Sep 05 06:55:33.646117 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:33.646066 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfdc139d06f8ad8012a5b58af70925d6d9361a563f61ba26f10205e5a99dac52"} err="failed to get container status \"dfdc139d06f8ad8012a5b58af70925d6d9361a563f61ba26f10205e5a99dac52\": rpc error: code = NotFound desc = could not find container \"dfdc139d06f8ad8012a5b58af70925d6d9361a563f61ba26f10205e5a99dac52\": container with ID starting with dfdc139d06f8ad8012a5b58af70925d6d9361a563f61ba26f10205e5a99dac52 not found: ID does not exist" Sep 05 06:55:33.651665 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:33.651646 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd"] Sep 05 06:55:33.656303 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:33.656281 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-183ae6-8697f989cf-phvrd"] Sep 05 06:55:33.665953 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:33.665935 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq"] Sep 05 06:55:33.670046 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:33.670030 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-183ae6-67b5bf86b-n56nq"] Sep 05 06:55:34.165260 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:34.165229 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="250ab2e7-8101-463f-8dee-591d425a1469" path="/var/lib/kubelet/pods/250ab2e7-8101-463f-8dee-591d425a1469/volumes" Sep 05 06:55:34.165562 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:34.165548 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f8b6536-fb94-4ff8-b811-c5b25fffbb54" path="/var/lib/kubelet/pods/7f8b6536-fb94-4ff8-b811-c5b25fffbb54/volumes" Sep 05 06:55:34.165822 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:34.165811 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b69b6519-cb9a-482a-8fbd-902283fe6a42" path="/var/lib/kubelet/pods/b69b6519-cb9a-482a-8fbd-902283fe6a42/volumes" Sep 05 06:55:34.166082 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:34.166071 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8160622-0e2a-431a-9952-7bdf6c0cc373" path="/var/lib/kubelet/pods/d8160622-0e2a-431a-9952-7bdf6c0cc373/volumes" Sep 05 06:55:39.735950 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:39.735897 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg"] Sep 05 06:55:39.736502 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:39.736177 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" podUID="b89fc0ad-b6f9-4cdd-8786-24883ebac401" containerName="istio-proxy" containerID="cri-o://ef55def92258fdfdfc00e2591cb204b8cd34ab219e27901b177e81a44e9dc896" gracePeriod=30 Sep 05 06:55:40.083662 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:40.083583 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-39047b-x25bz"] Sep 05 06:55:40.083999 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:40.083986 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b69b6519-cb9a-482a-8fbd-902283fe6a42" containerName="payload-processing" Sep 05 06:55:40.084047 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:40.084001 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b69b6519-cb9a-482a-8fbd-902283fe6a42" containerName="payload-processing" Sep 05 06:55:40.084047 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:40.084010 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="250ab2e7-8101-463f-8dee-591d425a1469" containerName="payload-pre-processing" Sep 05 06:55:40.084047 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:40.084015 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="250ab2e7-8101-463f-8dee-591d425a1469" containerName="payload-pre-processing" Sep 05 06:55:40.084047 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:40.084026 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8160622-0e2a-431a-9952-7bdf6c0cc373" containerName="revoke-keys" Sep 05 06:55:40.084047 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:40.084031 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8160622-0e2a-431a-9952-7bdf6c0cc373" containerName="revoke-keys" Sep 05 06:55:40.084198 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:40.084082 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8160622-0e2a-431a-9952-7bdf6c0cc373" containerName="revoke-keys" Sep 05 06:55:40.084198 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:40.084092 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="250ab2e7-8101-463f-8dee-591d425a1469" containerName="payload-pre-processing" Sep 05 06:55:40.084198 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:40.084097 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b69b6519-cb9a-482a-8fbd-902283fe6a42" containerName="payload-processing" Sep 05 06:55:40.088567 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:40.088550 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-39047b-x25bz" Sep 05 06:55:40.090912 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:40.090893 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 05 06:55:40.091046 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:40.090928 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slfqj\"" Sep 05 06:55:40.095273 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:40.095253 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-39047b-x25bz"] Sep 05 06:55:40.115215 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:40.115190 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b46206-632b-416e-8143-83d5d136c684-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-39047b-x25bz\" (UID: \"65b46206-632b-416e-8143-83d5d136c684\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-39047b-x25bz" Sep 05 06:55:40.216602 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:40.216574 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b46206-632b-416e-8143-83d5d136c684-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-39047b-x25bz\" (UID: \"65b46206-632b-416e-8143-83d5d136c684\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-39047b-x25bz" Sep 05 06:55:40.217156 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:40.217138 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b46206-632b-416e-8143-83d5d136c684-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-39047b-x25bz\" (UID: \"65b46206-632b-416e-8143-83d5d136c684\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-39047b-x25bz" Sep 05 06:55:40.401081 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:40.401037 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-39047b-x25bz" Sep 05 06:55:40.674742 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:40.674713 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-39047b-x25bz"] Sep 05 06:55:40.677365 ip-10-0-141-66 kubenswrapper[2565]: W0905 06:55:40.677302 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b46206_632b_416e_8143_83d5d136c684.slice/crio-da841c2589a971a90946a8761d8160cc3db51c3040ad9f30ed821f0994850cb1 WatchSource:0}: Error finding container da841c2589a971a90946a8761d8160cc3db51c3040ad9f30ed821f0994850cb1: Status 404 returned error can't find the container with id da841c2589a971a90946a8761d8160cc3db51c3040ad9f30ed821f0994850cb1 Sep 05 06:55:41.673419 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:41.673388 2565 generic.go:356] "Generic (PLEG): container finished" podID="65b46206-632b-416e-8143-83d5d136c684" containerID="c31caccbc61e34f5dd71c727374bdde83821cbfa2e298cfeba1813339212f959" exitCode=0 Sep 05 06:55:41.673783 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:41.673477 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-39047b-x25bz" event={"ID":"65b46206-632b-416e-8143-83d5d136c684","Type":"ContainerDied","Data":"c31caccbc61e34f5dd71c727374bdde83821cbfa2e298cfeba1813339212f959"} Sep 05 06:55:41.673783 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:41.673524 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-39047b-x25bz" event={"ID":"65b46206-632b-416e-8143-83d5d136c684","Type":"ContainerStarted","Data":"da841c2589a971a90946a8761d8160cc3db51c3040ad9f30ed821f0994850cb1"} Sep 05 06:55:41.881061 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:41.881027 2565 patch_prober.go:28] interesting pod/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 05 06:55:41.881211 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:41.881077 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" podUID="b89fc0ad-b6f9-4cdd-8786-24883ebac401" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 05 06:55:42.810934 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:42.810912 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-39047b-x25bz" Sep 05 06:55:42.841715 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:42.841689 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b46206-632b-416e-8143-83d5d136c684-maas-api-service-ca\") pod \"65b46206-632b-416e-8143-83d5d136c684\" (UID: \"65b46206-632b-416e-8143-83d5d136c684\") " Sep 05 06:55:42.842005 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:42.841984 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65b46206-632b-416e-8143-83d5d136c684-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "65b46206-632b-416e-8143-83d5d136c684" (UID: "65b46206-632b-416e-8143-83d5d136c684"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:55:42.943145 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:42.943072 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b46206-632b-416e-8143-83d5d136c684-maas-api-service-ca\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:43.684948 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:43.684913 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-39047b-x25bz" Sep 05 06:55:43.685140 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:43.684943 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-39047b-x25bz" event={"ID":"65b46206-632b-416e-8143-83d5d136c684","Type":"ContainerDied","Data":"da841c2589a971a90946a8761d8160cc3db51c3040ad9f30ed821f0994850cb1"} Sep 05 06:55:43.685140 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:43.684990 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da841c2589a971a90946a8761d8160cc3db51c3040ad9f30ed821f0994850cb1" Sep 05 06:55:44.992644 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:44.992617 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:55:45.063124 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.063046 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-workload-certs\") pod \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " Sep 05 06:55:45.063124 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.063093 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istio-data\") pod \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " Sep 05 06:55:45.063124 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.063122 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istio-podinfo\") pod \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " Sep 05 06:55:45.063427 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.063151 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istio-token\") pod \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " Sep 05 06:55:45.063427 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.063170 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istiod-ca-cert\") pod \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " Sep 05 06:55:45.063427 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.063226 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-workload-socket\") pod \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " Sep 05 06:55:45.063427 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.063264 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istio-envoy\") pod \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " Sep 05 06:55:45.063427 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.063289 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-credential-socket\") pod \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " Sep 05 06:55:45.063427 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.063359 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b89fc0ad-b6f9-4cdd-8786-24883ebac401" (UID: "b89fc0ad-b6f9-4cdd-8786-24883ebac401"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:55:45.063427 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.063399 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7r5s\" (UniqueName: \"kubernetes.io/projected/b89fc0ad-b6f9-4cdd-8786-24883ebac401-kube-api-access-c7r5s\") pod \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\" (UID: \"b89fc0ad-b6f9-4cdd-8786-24883ebac401\") " Sep 05 06:55:45.063779 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.063586 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b89fc0ad-b6f9-4cdd-8786-24883ebac401" (UID: "b89fc0ad-b6f9-4cdd-8786-24883ebac401"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:55:45.063779 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.063636 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b89fc0ad-b6f9-4cdd-8786-24883ebac401" (UID: "b89fc0ad-b6f9-4cdd-8786-24883ebac401"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:55:45.063779 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.063702 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b89fc0ad-b6f9-4cdd-8786-24883ebac401" (UID: "b89fc0ad-b6f9-4cdd-8786-24883ebac401"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:55:45.063779 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.063538 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istio-data" (OuterVolumeSpecName: "istio-data") pod "b89fc0ad-b6f9-4cdd-8786-24883ebac401" (UID: "b89fc0ad-b6f9-4cdd-8786-24883ebac401"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:55:45.063779 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.063747 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-workload-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:45.063779 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.063766 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istiod-ca-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:45.063779 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.063778 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-workload-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:45.065630 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.065607 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b89fc0ad-b6f9-4cdd-8786-24883ebac401" (UID: "b89fc0ad-b6f9-4cdd-8786-24883ebac401"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:55:45.065630 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.065616 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istio-token" (OuterVolumeSpecName: "istio-token") pod "b89fc0ad-b6f9-4cdd-8786-24883ebac401" (UID: "b89fc0ad-b6f9-4cdd-8786-24883ebac401"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:55:45.065781 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.065681 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b89fc0ad-b6f9-4cdd-8786-24883ebac401" (UID: "b89fc0ad-b6f9-4cdd-8786-24883ebac401"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 05 06:55:45.065781 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.065738 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b89fc0ad-b6f9-4cdd-8786-24883ebac401-kube-api-access-c7r5s" (OuterVolumeSpecName: "kube-api-access-c7r5s") pod "b89fc0ad-b6f9-4cdd-8786-24883ebac401" (UID: "b89fc0ad-b6f9-4cdd-8786-24883ebac401"). InnerVolumeSpecName "kube-api-access-c7r5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:55:45.165163 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.165129 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istio-envoy\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:45.165163 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.165157 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-credential-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:45.165163 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.165168 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7r5s\" (UniqueName: \"kubernetes.io/projected/b89fc0ad-b6f9-4cdd-8786-24883ebac401-kube-api-access-c7r5s\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:45.165435 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.165177 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istio-data\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:45.165435 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.165185 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istio-podinfo\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:45.165435 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.165192 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b89fc0ad-b6f9-4cdd-8786-24883ebac401-istio-token\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:55:45.699411 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.699375 2565 generic.go:356] "Generic (PLEG): container finished" podID="b89fc0ad-b6f9-4cdd-8786-24883ebac401" containerID="ef55def92258fdfdfc00e2591cb204b8cd34ab219e27901b177e81a44e9dc896" exitCode=0 Sep 05 06:55:45.699562 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.699432 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" Sep 05 06:55:45.699562 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.699453 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" event={"ID":"b89fc0ad-b6f9-4cdd-8786-24883ebac401","Type":"ContainerDied","Data":"ef55def92258fdfdfc00e2591cb204b8cd34ab219e27901b177e81a44e9dc896"} Sep 05 06:55:45.699562 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.699508 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg" event={"ID":"b89fc0ad-b6f9-4cdd-8786-24883ebac401","Type":"ContainerDied","Data":"d83d63c54af176e6ab4739a2412dacf6a842be1ac1429814452701bd7f54c58a"} Sep 05 06:55:45.699562 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.699532 2565 scope.go:117] "RemoveContainer" containerID="ef55def92258fdfdfc00e2591cb204b8cd34ab219e27901b177e81a44e9dc896" Sep 05 06:55:45.709073 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.709059 2565 scope.go:117] "RemoveContainer" containerID="7f27a85910310ecc02b034d4a31bea15c2696681fa809759c7c13f8167a5309a" Sep 05 06:55:45.717091 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.717077 2565 scope.go:117] "RemoveContainer" containerID="ef55def92258fdfdfc00e2591cb204b8cd34ab219e27901b177e81a44e9dc896" Sep 05 06:55:45.717331 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:55:45.717298 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef55def92258fdfdfc00e2591cb204b8cd34ab219e27901b177e81a44e9dc896\": container with ID starting with ef55def92258fdfdfc00e2591cb204b8cd34ab219e27901b177e81a44e9dc896 not found: ID does not exist" containerID="ef55def92258fdfdfc00e2591cb204b8cd34ab219e27901b177e81a44e9dc896" Sep 05 06:55:45.717410 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.717338 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef55def92258fdfdfc00e2591cb204b8cd34ab219e27901b177e81a44e9dc896"} err="failed to get container status \"ef55def92258fdfdfc00e2591cb204b8cd34ab219e27901b177e81a44e9dc896\": rpc error: code = NotFound desc = could not find container \"ef55def92258fdfdfc00e2591cb204b8cd34ab219e27901b177e81a44e9dc896\": container with ID starting with ef55def92258fdfdfc00e2591cb204b8cd34ab219e27901b177e81a44e9dc896 not found: ID does not exist" Sep 05 06:55:45.717410 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.717355 2565 scope.go:117] "RemoveContainer" containerID="7f27a85910310ecc02b034d4a31bea15c2696681fa809759c7c13f8167a5309a" Sep 05 06:55:45.717595 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:55:45.717576 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f27a85910310ecc02b034d4a31bea15c2696681fa809759c7c13f8167a5309a\": container with ID starting with 7f27a85910310ecc02b034d4a31bea15c2696681fa809759c7c13f8167a5309a not found: ID does not exist" containerID="7f27a85910310ecc02b034d4a31bea15c2696681fa809759c7c13f8167a5309a" Sep 05 06:55:45.717644 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.717600 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f27a85910310ecc02b034d4a31bea15c2696681fa809759c7c13f8167a5309a"} err="failed to get container status \"7f27a85910310ecc02b034d4a31bea15c2696681fa809759c7c13f8167a5309a\": rpc error: code = NotFound desc = could not find container \"7f27a85910310ecc02b034d4a31bea15c2696681fa809759c7c13f8167a5309a\": container with ID starting with 7f27a85910310ecc02b034d4a31bea15c2696681fa809759c7c13f8167a5309a not found: ID does not exist" Sep 05 06:55:45.721647 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.721623 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg"] Sep 05 06:55:45.725137 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:45.725111 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-183ae6-openshift-default-6f84bb458d-5sbdg"] Sep 05 06:55:46.165888 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:46.165856 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b89fc0ad-b6f9-4cdd-8786-24883ebac401" path="/var/lib/kubelet/pods/b89fc0ad-b6f9-4cdd-8786-24883ebac401/volumes" Sep 05 06:55:51.085759 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:51.085724 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-39047b-x25bz"] Sep 05 06:55:51.091149 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:51.091120 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-39047b-x25bz"] Sep 05 06:55:52.165544 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:52.165505 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65b46206-632b-416e-8143-83d5d136c684" path="/var/lib/kubelet/pods/65b46206-632b-416e-8143-83d5d136c684/volumes" Sep 05 06:55:58.031876 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:58.031843 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx"] Sep 05 06:55:58.032352 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:55:58.032066 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" podUID="ec947db4-e6a6-4f5e-af7d-58712a785965" containerName="istio-proxy" containerID="cri-o://71d5a2550c5ca2cd73b7eec2baae51144536d19a91e5b2dd646141c2a1af9d77" gracePeriod=30 Sep 05 06:56:03.284072 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.284042 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:56:03.430568 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.430539 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-credential-socket\") pod \"ec947db4-e6a6-4f5e-af7d-58712a785965\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " Sep 05 06:56:03.430726 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.430578 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec947db4-e6a6-4f5e-af7d-58712a785965-istio-token\") pod \"ec947db4-e6a6-4f5e-af7d-58712a785965\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " Sep 05 06:56:03.430726 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.430612 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-workload-socket\") pod \"ec947db4-e6a6-4f5e-af7d-58712a785965\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " Sep 05 06:56:03.430726 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.430665 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec947db4-e6a6-4f5e-af7d-58712a785965-istio-podinfo\") pod \"ec947db4-e6a6-4f5e-af7d-58712a785965\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " Sep 05 06:56:03.430726 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.430681 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-workload-certs\") pod \"ec947db4-e6a6-4f5e-af7d-58712a785965\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " Sep 05 06:56:03.430726 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.430715 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-istio-data\") pod \"ec947db4-e6a6-4f5e-af7d-58712a785965\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " Sep 05 06:56:03.430999 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.430752 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec947db4-e6a6-4f5e-af7d-58712a785965-istiod-ca-cert\") pod \"ec947db4-e6a6-4f5e-af7d-58712a785965\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " Sep 05 06:56:03.430999 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.430785 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-istio-envoy\") pod \"ec947db4-e6a6-4f5e-af7d-58712a785965\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " Sep 05 06:56:03.430999 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.430825 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26nn5\" (UniqueName: \"kubernetes.io/projected/ec947db4-e6a6-4f5e-af7d-58712a785965-kube-api-access-26nn5\") pod \"ec947db4-e6a6-4f5e-af7d-58712a785965\" (UID: \"ec947db4-e6a6-4f5e-af7d-58712a785965\") " Sep 05 06:56:03.430999 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.430974 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ec947db4-e6a6-4f5e-af7d-58712a785965" (UID: "ec947db4-e6a6-4f5e-af7d-58712a785965"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:56:03.431192 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.431006 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ec947db4-e6a6-4f5e-af7d-58712a785965" (UID: "ec947db4-e6a6-4f5e-af7d-58712a785965"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:56:03.431192 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.431071 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-istio-data" (OuterVolumeSpecName: "istio-data") pod "ec947db4-e6a6-4f5e-af7d-58712a785965" (UID: "ec947db4-e6a6-4f5e-af7d-58712a785965"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:56:03.431334 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.431202 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec947db4-e6a6-4f5e-af7d-58712a785965-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ec947db4-e6a6-4f5e-af7d-58712a785965" (UID: "ec947db4-e6a6-4f5e-af7d-58712a785965"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 06:56:03.431792 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.431652 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-workload-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:56:03.431792 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.431677 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-workload-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:56:03.431792 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.431692 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-istio-data\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:56:03.431792 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.431705 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec947db4-e6a6-4f5e-af7d-58712a785965-istiod-ca-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:56:03.431792 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.431764 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ec947db4-e6a6-4f5e-af7d-58712a785965" (UID: "ec947db4-e6a6-4f5e-af7d-58712a785965"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:56:03.433010 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.432982 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec947db4-e6a6-4f5e-af7d-58712a785965-istio-token" (OuterVolumeSpecName: "istio-token") pod "ec947db4-e6a6-4f5e-af7d-58712a785965" (UID: "ec947db4-e6a6-4f5e-af7d-58712a785965"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:56:03.433010 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.432984 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ec947db4-e6a6-4f5e-af7d-58712a785965" (UID: "ec947db4-e6a6-4f5e-af7d-58712a785965"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 06:56:03.433237 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.433218 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec947db4-e6a6-4f5e-af7d-58712a785965-kube-api-access-26nn5" (OuterVolumeSpecName: "kube-api-access-26nn5") pod "ec947db4-e6a6-4f5e-af7d-58712a785965" (UID: "ec947db4-e6a6-4f5e-af7d-58712a785965"). InnerVolumeSpecName "kube-api-access-26nn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 06:56:03.433359 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.433342 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec947db4-e6a6-4f5e-af7d-58712a785965-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ec947db4-e6a6-4f5e-af7d-58712a785965" (UID: "ec947db4-e6a6-4f5e-af7d-58712a785965"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 05 06:56:03.532990 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.532963 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec947db4-e6a6-4f5e-af7d-58712a785965-istio-podinfo\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:56:03.532990 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.532987 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-istio-envoy\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:56:03.532990 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.532997 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26nn5\" (UniqueName: \"kubernetes.io/projected/ec947db4-e6a6-4f5e-af7d-58712a785965-kube-api-access-26nn5\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:56:03.533203 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.533007 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec947db4-e6a6-4f5e-af7d-58712a785965-credential-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:56:03.533203 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.533015 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec947db4-e6a6-4f5e-af7d-58712a785965-istio-token\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 06:56:03.804055 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.803958 2565 generic.go:356] "Generic (PLEG): container finished" podID="ec947db4-e6a6-4f5e-af7d-58712a785965" containerID="71d5a2550c5ca2cd73b7eec2baae51144536d19a91e5b2dd646141c2a1af9d77" exitCode=0 Sep 05 06:56:03.804055 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.804017 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" Sep 05 06:56:03.804232 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.804078 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" event={"ID":"ec947db4-e6a6-4f5e-af7d-58712a785965","Type":"ContainerDied","Data":"71d5a2550c5ca2cd73b7eec2baae51144536d19a91e5b2dd646141c2a1af9d77"} Sep 05 06:56:03.804232 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.804148 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx" event={"ID":"ec947db4-e6a6-4f5e-af7d-58712a785965","Type":"ContainerDied","Data":"7c5f9f14c30c682ef61319e6cbd16982c709ad2402a5e7f299655cc34efbd0ac"} Sep 05 06:56:03.804232 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.804170 2565 scope.go:117] "RemoveContainer" containerID="71d5a2550c5ca2cd73b7eec2baae51144536d19a91e5b2dd646141c2a1af9d77" Sep 05 06:56:03.813513 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.813496 2565 scope.go:117] "RemoveContainer" containerID="3602b5519f99ad970f394f829fd75e8448e40bd5e66d4710c8f73cbbec1d3efb" Sep 05 06:56:03.821221 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.821201 2565 scope.go:117] "RemoveContainer" containerID="71d5a2550c5ca2cd73b7eec2baae51144536d19a91e5b2dd646141c2a1af9d77" Sep 05 06:56:03.821548 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:56:03.821528 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d5a2550c5ca2cd73b7eec2baae51144536d19a91e5b2dd646141c2a1af9d77\": container with ID starting with 71d5a2550c5ca2cd73b7eec2baae51144536d19a91e5b2dd646141c2a1af9d77 not found: ID does not exist" containerID="71d5a2550c5ca2cd73b7eec2baae51144536d19a91e5b2dd646141c2a1af9d77" Sep 05 06:56:03.821639 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.821553 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d5a2550c5ca2cd73b7eec2baae51144536d19a91e5b2dd646141c2a1af9d77"} err="failed to get container status \"71d5a2550c5ca2cd73b7eec2baae51144536d19a91e5b2dd646141c2a1af9d77\": rpc error: code = NotFound desc = could not find container \"71d5a2550c5ca2cd73b7eec2baae51144536d19a91e5b2dd646141c2a1af9d77\": container with ID starting with 71d5a2550c5ca2cd73b7eec2baae51144536d19a91e5b2dd646141c2a1af9d77 not found: ID does not exist" Sep 05 06:56:03.821639 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.821570 2565 scope.go:117] "RemoveContainer" containerID="3602b5519f99ad970f394f829fd75e8448e40bd5e66d4710c8f73cbbec1d3efb" Sep 05 06:56:03.821848 ip-10-0-141-66 kubenswrapper[2565]: E0905 06:56:03.821830 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3602b5519f99ad970f394f829fd75e8448e40bd5e66d4710c8f73cbbec1d3efb\": container with ID starting with 3602b5519f99ad970f394f829fd75e8448e40bd5e66d4710c8f73cbbec1d3efb not found: ID does not exist" containerID="3602b5519f99ad970f394f829fd75e8448e40bd5e66d4710c8f73cbbec1d3efb" Sep 05 06:56:03.821883 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.821856 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3602b5519f99ad970f394f829fd75e8448e40bd5e66d4710c8f73cbbec1d3efb"} err="failed to get container status \"3602b5519f99ad970f394f829fd75e8448e40bd5e66d4710c8f73cbbec1d3efb\": rpc error: code = NotFound desc = could not find container \"3602b5519f99ad970f394f829fd75e8448e40bd5e66d4710c8f73cbbec1d3efb\": container with ID starting with 3602b5519f99ad970f394f829fd75e8448e40bd5e66d4710c8f73cbbec1d3efb not found: ID does not exist" Sep 05 06:56:03.826195 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.826173 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx"] Sep 05 06:56:03.829923 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:03.829903 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-39047b-openshift-default-546bffdc64-k88gx"] Sep 05 06:56:04.165597 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:04.165569 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec947db4-e6a6-4f5e-af7d-58712a785965" path="/var/lib/kubelet/pods/ec947db4-e6a6-4f5e-af7d-58712a785965/volumes" Sep 05 06:56:16.501690 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:56:16.501659 2565 scope.go:117] "RemoveContainer" containerID="175c677ee1ed1427e6e502728d81f2ed176f91a12c2094abfb33ad355240d459" Sep 05 06:57:16.537919 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:57:16.537890 2565 scope.go:117] "RemoveContainer" containerID="c183d35a0aacb3d13df6a68bf04f870a61fcbfb4ca81ec8899dbbf88417fefe6" Sep 05 06:57:16.546351 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:57:16.546329 2565 scope.go:117] "RemoveContainer" containerID="bda61ad7dfe951c480dbd85fe1488b9a22c23d951bb74de9ded4440af69c9e5a" Sep 05 06:58:16.570040 ip-10-0-141-66 kubenswrapper[2565]: I0905 06:58:16.570007 2565 scope.go:117] "RemoveContainer" containerID="9bc0f56ab8bcd7e491f4d94f5d51e201bdc863c80efa8fd03b14d816e950e0f6" Sep 05 07:00:00.177325 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:00.177280 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29809860-pbf97"] Sep 05 07:00:00.177823 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:00.177643 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65b46206-632b-416e-8143-83d5d136c684" containerName="revoke-keys" Sep 05 07:00:00.177823 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:00.177654 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="65b46206-632b-416e-8143-83d5d136c684" containerName="revoke-keys" Sep 05 07:00:00.177823 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:00.177667 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b89fc0ad-b6f9-4cdd-8786-24883ebac401" containerName="istio-proxy" Sep 05 07:00:00.177823 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:00.177673 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b89fc0ad-b6f9-4cdd-8786-24883ebac401" containerName="istio-proxy" Sep 05 07:00:00.177823 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:00.177684 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec947db4-e6a6-4f5e-af7d-58712a785965" containerName="istio-proxy" Sep 05 07:00:00.177823 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:00.177689 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec947db4-e6a6-4f5e-af7d-58712a785965" containerName="istio-proxy" Sep 05 07:00:00.177823 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:00.177699 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec947db4-e6a6-4f5e-af7d-58712a785965" containerName="istio-proxy" Sep 05 07:00:00.177823 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:00.177704 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec947db4-e6a6-4f5e-af7d-58712a785965" containerName="istio-proxy" Sep 05 07:00:00.177823 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:00.177761 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b89fc0ad-b6f9-4cdd-8786-24883ebac401" containerName="istio-proxy" Sep 05 07:00:00.177823 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:00.177769 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec947db4-e6a6-4f5e-af7d-58712a785965" containerName="istio-proxy" Sep 05 07:00:00.177823 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:00.177774 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec947db4-e6a6-4f5e-af7d-58712a785965" containerName="istio-proxy" Sep 05 07:00:00.177823 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:00.177780 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="65b46206-632b-416e-8143-83d5d136c684" containerName="revoke-keys" Sep 05 07:00:00.177823 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:00.177787 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b89fc0ad-b6f9-4cdd-8786-24883ebac401" containerName="istio-proxy" Sep 05 07:00:00.180831 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:00.180803 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809860-pbf97" Sep 05 07:00:00.184998 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:00.184978 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slfqj\"" Sep 05 07:00:00.190146 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:00.190122 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809860-pbf97" Sep 05 07:00:00.192850 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:00.192766 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29809860-pbf97"] Sep 05 07:00:00.463987 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:00.463961 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29809860-pbf97"] Sep 05 07:00:00.466077 ip-10-0-141-66 kubenswrapper[2565]: W0905 07:00:00.466048 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941e07e0_fb08_4f2b_93f1_ce5f0c41bb27.slice/crio-a08413778df13defd498725fcfc2d7f05729ae140dd98014484b47e17e3f6d8e WatchSource:0}: Error finding container a08413778df13defd498725fcfc2d7f05729ae140dd98014484b47e17e3f6d8e: Status 404 returned error can't find the container with id a08413778df13defd498725fcfc2d7f05729ae140dd98014484b47e17e3f6d8e Sep 05 07:00:00.958084 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:00.958051 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809860-pbf97" event={"ID":"941e07e0-fb08-4f2b-93f1-ce5f0c41bb27","Type":"ContainerStarted","Data":"319021b0c2600b25f9dfe4b8a9d6a5da7049315d9748554ddf6e56a2ed3264fd"} Sep 05 07:00:00.958349 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:00.958110 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809860-pbf97" event={"ID":"941e07e0-fb08-4f2b-93f1-ce5f0c41bb27","Type":"ContainerStarted","Data":"a08413778df13defd498725fcfc2d7f05729ae140dd98014484b47e17e3f6d8e"} Sep 05 07:00:00.973521 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:00.973477 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809860-pbf97" podStartSLOduration=0.973460451 podStartE2EDuration="973.460451ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:00:00.972692151 +0000 UTC m=+2025.357235276" watchObservedRunningTime="2026-09-05 07:00:00.973460451 +0000 UTC m=+2025.358003574" Sep 05 07:00:01.964111 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:01.964071 2565 generic.go:356] "Generic (PLEG): container finished" podID="941e07e0-fb08-4f2b-93f1-ce5f0c41bb27" containerID="319021b0c2600b25f9dfe4b8a9d6a5da7049315d9748554ddf6e56a2ed3264fd" exitCode=0 Sep 05 07:00:01.964522 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:01.964140 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809860-pbf97" event={"ID":"941e07e0-fb08-4f2b-93f1-ce5f0c41bb27","Type":"ContainerDied","Data":"319021b0c2600b25f9dfe4b8a9d6a5da7049315d9748554ddf6e56a2ed3264fd"} Sep 05 07:00:03.098232 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:03.098209 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809860-pbf97" Sep 05 07:00:03.975401 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:03.975369 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809860-pbf97" Sep 05 07:00:03.975401 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:03.975378 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809860-pbf97" event={"ID":"941e07e0-fb08-4f2b-93f1-ce5f0c41bb27","Type":"ContainerDied","Data":"a08413778df13defd498725fcfc2d7f05729ae140dd98014484b47e17e3f6d8e"} Sep 05 07:00:03.975673 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:03.975442 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a08413778df13defd498725fcfc2d7f05729ae140dd98014484b47e17e3f6d8e" Sep 05 07:00:16.596651 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:00:16.596619 2565 scope.go:117] "RemoveContainer" containerID="864e1da46a646a487d1a2a6333594b6a8abdd1eb61ddbca94ee449c701e5e111" Sep 05 07:01:09.992356 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:09.992293 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw"] Sep 05 07:01:09.992926 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:09.992892 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b89fc0ad-b6f9-4cdd-8786-24883ebac401" containerName="istio-proxy" Sep 05 07:01:09.992926 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:09.992912 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b89fc0ad-b6f9-4cdd-8786-24883ebac401" containerName="istio-proxy" Sep 05 07:01:09.992926 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:09.992927 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="941e07e0-fb08-4f2b-93f1-ce5f0c41bb27" containerName="cleanup" Sep 05 07:01:09.993119 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:09.992936 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="941e07e0-fb08-4f2b-93f1-ce5f0c41bb27" containerName="cleanup" Sep 05 07:01:09.993119 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:09.993037 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="941e07e0-fb08-4f2b-93f1-ce5f0c41bb27" containerName="cleanup" Sep 05 07:01:09.996192 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:09.996169 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw" Sep 05 07:01:09.999174 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:09.999148 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 05 07:01:09.999174 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:09.999159 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 05 07:01:09.999381 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:09.999148 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 05 07:01:09.999381 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:09.999148 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qdhh9\"" Sep 05 07:01:09.999381 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:09.999149 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 05 07:01:10.006050 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:10.006031 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw"] Sep 05 07:01:10.081256 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:10.081223 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rm5f\" (UniqueName: \"kubernetes.io/projected/87d08375-dbed-4a70-83e1-2481b7754ae5-kube-api-access-8rm5f\") pod \"maas-api-8457d65c99-9vrbw\" (UID: \"87d08375-dbed-4a70-83e1-2481b7754ae5\") " pod="odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw" Sep 05 07:01:10.081475 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:10.081269 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87d08375-dbed-4a70-83e1-2481b7754ae5-maas-api-tls\") pod \"maas-api-8457d65c99-9vrbw\" (UID: \"87d08375-dbed-4a70-83e1-2481b7754ae5\") " pod="odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw" Sep 05 07:01:10.081475 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:10.081290 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87d08375-dbed-4a70-83e1-2481b7754ae5-openshift-service-ca\") pod \"maas-api-8457d65c99-9vrbw\" (UID: \"87d08375-dbed-4a70-83e1-2481b7754ae5\") " pod="odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw" Sep 05 07:01:10.081475 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:10.081432 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87d08375-dbed-4a70-83e1-2481b7754ae5-metrics-certs\") pod \"maas-api-8457d65c99-9vrbw\" (UID: \"87d08375-dbed-4a70-83e1-2481b7754ae5\") " pod="odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw" Sep 05 07:01:10.182480 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:10.182450 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rm5f\" (UniqueName: \"kubernetes.io/projected/87d08375-dbed-4a70-83e1-2481b7754ae5-kube-api-access-8rm5f\") pod \"maas-api-8457d65c99-9vrbw\" (UID: \"87d08375-dbed-4a70-83e1-2481b7754ae5\") " pod="odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw" Sep 05 07:01:10.182656 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:10.182494 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87d08375-dbed-4a70-83e1-2481b7754ae5-maas-api-tls\") pod \"maas-api-8457d65c99-9vrbw\" (UID: \"87d08375-dbed-4a70-83e1-2481b7754ae5\") " pod="odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw" Sep 05 07:01:10.182656 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:10.182531 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87d08375-dbed-4a70-83e1-2481b7754ae5-openshift-service-ca\") pod \"maas-api-8457d65c99-9vrbw\" (UID: \"87d08375-dbed-4a70-83e1-2481b7754ae5\") " pod="odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw" Sep 05 07:01:10.182656 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:10.182584 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87d08375-dbed-4a70-83e1-2481b7754ae5-metrics-certs\") pod \"maas-api-8457d65c99-9vrbw\" (UID: \"87d08375-dbed-4a70-83e1-2481b7754ae5\") " pod="odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw" Sep 05 07:01:10.183202 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:10.183174 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87d08375-dbed-4a70-83e1-2481b7754ae5-openshift-service-ca\") pod \"maas-api-8457d65c99-9vrbw\" (UID: \"87d08375-dbed-4a70-83e1-2481b7754ae5\") " pod="odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw" Sep 05 07:01:10.185030 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:10.185011 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87d08375-dbed-4a70-83e1-2481b7754ae5-metrics-certs\") pod \"maas-api-8457d65c99-9vrbw\" (UID: \"87d08375-dbed-4a70-83e1-2481b7754ae5\") " pod="odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw" Sep 05 07:01:10.185127 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:10.185072 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87d08375-dbed-4a70-83e1-2481b7754ae5-maas-api-tls\") pod \"maas-api-8457d65c99-9vrbw\" (UID: \"87d08375-dbed-4a70-83e1-2481b7754ae5\") " pod="odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw" Sep 05 07:01:10.191338 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:10.191288 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rm5f\" (UniqueName: \"kubernetes.io/projected/87d08375-dbed-4a70-83e1-2481b7754ae5-kube-api-access-8rm5f\") pod \"maas-api-8457d65c99-9vrbw\" (UID: \"87d08375-dbed-4a70-83e1-2481b7754ae5\") " pod="odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw" Sep 05 07:01:10.309012 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:10.308926 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw" Sep 05 07:01:10.585659 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:10.585633 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw"] Sep 05 07:01:10.588271 ip-10-0-141-66 kubenswrapper[2565]: W0905 07:01:10.588228 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d08375_dbed_4a70_83e1_2481b7754ae5.slice/crio-81796685e830103f2c1c52112cc5810fdb477453bf0f4ca00a6965369906f912 WatchSource:0}: Error finding container 81796685e830103f2c1c52112cc5810fdb477453bf0f4ca00a6965369906f912: Status 404 returned error can't find the container with id 81796685e830103f2c1c52112cc5810fdb477453bf0f4ca00a6965369906f912 Sep 05 07:01:10.589682 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:10.589662 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 05 07:01:11.297489 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:11.297454 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw" event={"ID":"87d08375-dbed-4a70-83e1-2481b7754ae5","Type":"ContainerStarted","Data":"81796685e830103f2c1c52112cc5810fdb477453bf0f4ca00a6965369906f912"} Sep 05 07:01:13.311001 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:13.310961 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw" event={"ID":"87d08375-dbed-4a70-83e1-2481b7754ae5","Type":"ContainerStarted","Data":"4f3b0e1bb317f963fcea45a3c3238d987836f4add463bd75734d62571ebbd8d4"} Sep 05 07:01:13.311412 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:13.311025 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw" Sep 05 07:01:13.332076 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:13.332013 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw" podStartSLOduration=2.110143904 podStartE2EDuration="4.331997412s" totalImagesPullingTime="2.221853508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:01:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:01:13.330780788 +0000 UTC m=+2097.715323913" watchObservedRunningTime="2026-09-05 07:01:13.331997412 +0000 UTC m=+2097.716540536" Sep 05 07:01:17.731279 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:17.731242 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw"] Sep 05 07:01:17.731838 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:17.731515 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw" podUID="87d08375-dbed-4a70-83e1-2481b7754ae5" containerName="maas-api" containerID="cri-o://4f3b0e1bb317f963fcea45a3c3238d987836f4add463bd75734d62571ebbd8d4" gracePeriod=30 Sep 05 07:01:17.736780 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:17.736757 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw" Sep 05 07:01:17.762384 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:17.762357 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-c6866d55f-7zvdq"] Sep 05 07:01:17.765640 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:17.765620 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c6866d55f-7zvdq" Sep 05 07:01:17.775514 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:17.775486 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c6866d55f-7zvdq"] Sep 05 07:01:17.853770 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:17.853747 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zchzj\" (UniqueName: \"kubernetes.io/projected/8a022621-eb76-46ed-822f-ac4f70fd4b95-kube-api-access-zchzj\") pod \"maas-api-c6866d55f-7zvdq\" (UID: \"8a022621-eb76-46ed-822f-ac4f70fd4b95\") " pod="odh-ai-gateway-infra/maas-api-c6866d55f-7zvdq" Sep 05 07:01:17.853880 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:17.853807 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a022621-eb76-46ed-822f-ac4f70fd4b95-maas-api-tls\") pod \"maas-api-c6866d55f-7zvdq\" (UID: \"8a022621-eb76-46ed-822f-ac4f70fd4b95\") " pod="odh-ai-gateway-infra/maas-api-c6866d55f-7zvdq" Sep 05 07:01:17.853957 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:17.853929 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a022621-eb76-46ed-822f-ac4f70fd4b95-metrics-certs\") pod \"maas-api-c6866d55f-7zvdq\" (UID: \"8a022621-eb76-46ed-822f-ac4f70fd4b95\") " pod="odh-ai-gateway-infra/maas-api-c6866d55f-7zvdq" Sep 05 07:01:17.854039 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:17.854022 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a022621-eb76-46ed-822f-ac4f70fd4b95-openshift-service-ca\") pod \"maas-api-c6866d55f-7zvdq\" (UID: \"8a022621-eb76-46ed-822f-ac4f70fd4b95\") " pod="odh-ai-gateway-infra/maas-api-c6866d55f-7zvdq" Sep 05 07:01:17.955230 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:17.955198 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a022621-eb76-46ed-822f-ac4f70fd4b95-maas-api-tls\") pod \"maas-api-c6866d55f-7zvdq\" (UID: \"8a022621-eb76-46ed-822f-ac4f70fd4b95\") " pod="odh-ai-gateway-infra/maas-api-c6866d55f-7zvdq" Sep 05 07:01:17.955405 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:17.955271 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a022621-eb76-46ed-822f-ac4f70fd4b95-metrics-certs\") pod \"maas-api-c6866d55f-7zvdq\" (UID: \"8a022621-eb76-46ed-822f-ac4f70fd4b95\") " pod="odh-ai-gateway-infra/maas-api-c6866d55f-7zvdq" Sep 05 07:01:17.955458 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:17.955434 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a022621-eb76-46ed-822f-ac4f70fd4b95-openshift-service-ca\") pod \"maas-api-c6866d55f-7zvdq\" (UID: \"8a022621-eb76-46ed-822f-ac4f70fd4b95\") " pod="odh-ai-gateway-infra/maas-api-c6866d55f-7zvdq" Sep 05 07:01:17.955527 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:17.955513 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zchzj\" (UniqueName: \"kubernetes.io/projected/8a022621-eb76-46ed-822f-ac4f70fd4b95-kube-api-access-zchzj\") pod \"maas-api-c6866d55f-7zvdq\" (UID: \"8a022621-eb76-46ed-822f-ac4f70fd4b95\") " pod="odh-ai-gateway-infra/maas-api-c6866d55f-7zvdq" Sep 05 07:01:17.956261 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:17.956225 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a022621-eb76-46ed-822f-ac4f70fd4b95-openshift-service-ca\") pod \"maas-api-c6866d55f-7zvdq\" (UID: \"8a022621-eb76-46ed-822f-ac4f70fd4b95\") " pod="odh-ai-gateway-infra/maas-api-c6866d55f-7zvdq" Sep 05 07:01:17.957930 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:17.957910 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a022621-eb76-46ed-822f-ac4f70fd4b95-metrics-certs\") pod \"maas-api-c6866d55f-7zvdq\" (UID: \"8a022621-eb76-46ed-822f-ac4f70fd4b95\") " pod="odh-ai-gateway-infra/maas-api-c6866d55f-7zvdq" Sep 05 07:01:17.958054 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:17.958037 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a022621-eb76-46ed-822f-ac4f70fd4b95-maas-api-tls\") pod \"maas-api-c6866d55f-7zvdq\" (UID: \"8a022621-eb76-46ed-822f-ac4f70fd4b95\") " pod="odh-ai-gateway-infra/maas-api-c6866d55f-7zvdq" Sep 05 07:01:17.964426 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:17.964408 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zchzj\" (UniqueName: \"kubernetes.io/projected/8a022621-eb76-46ed-822f-ac4f70fd4b95-kube-api-access-zchzj\") pod \"maas-api-c6866d55f-7zvdq\" (UID: \"8a022621-eb76-46ed-822f-ac4f70fd4b95\") " pod="odh-ai-gateway-infra/maas-api-c6866d55f-7zvdq" Sep 05 07:01:17.983933 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:17.983880 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw" Sep 05 07:01:18.056593 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:18.056567 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87d08375-dbed-4a70-83e1-2481b7754ae5-openshift-service-ca\") pod \"87d08375-dbed-4a70-83e1-2481b7754ae5\" (UID: \"87d08375-dbed-4a70-83e1-2481b7754ae5\") " Sep 05 07:01:18.056751 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:18.056610 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87d08375-dbed-4a70-83e1-2481b7754ae5-metrics-certs\") pod \"87d08375-dbed-4a70-83e1-2481b7754ae5\" (UID: \"87d08375-dbed-4a70-83e1-2481b7754ae5\") " Sep 05 07:01:18.056751 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:18.056670 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rm5f\" (UniqueName: \"kubernetes.io/projected/87d08375-dbed-4a70-83e1-2481b7754ae5-kube-api-access-8rm5f\") pod \"87d08375-dbed-4a70-83e1-2481b7754ae5\" (UID: \"87d08375-dbed-4a70-83e1-2481b7754ae5\") " Sep 05 07:01:18.056751 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:18.056745 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87d08375-dbed-4a70-83e1-2481b7754ae5-maas-api-tls\") pod \"87d08375-dbed-4a70-83e1-2481b7754ae5\" (UID: \"87d08375-dbed-4a70-83e1-2481b7754ae5\") " Sep 05 07:01:18.056977 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:18.056953 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87d08375-dbed-4a70-83e1-2481b7754ae5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "87d08375-dbed-4a70-83e1-2481b7754ae5" (UID: "87d08375-dbed-4a70-83e1-2481b7754ae5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 07:01:18.057065 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:18.057022 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87d08375-dbed-4a70-83e1-2481b7754ae5-openshift-service-ca\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 07:01:18.059021 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:18.058990 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87d08375-dbed-4a70-83e1-2481b7754ae5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "87d08375-dbed-4a70-83e1-2481b7754ae5" (UID: "87d08375-dbed-4a70-83e1-2481b7754ae5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 07:01:18.059136 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:18.059038 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d08375-dbed-4a70-83e1-2481b7754ae5-kube-api-access-8rm5f" (OuterVolumeSpecName: "kube-api-access-8rm5f") pod "87d08375-dbed-4a70-83e1-2481b7754ae5" (UID: "87d08375-dbed-4a70-83e1-2481b7754ae5"). InnerVolumeSpecName "kube-api-access-8rm5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 07:01:18.059136 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:18.059071 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87d08375-dbed-4a70-83e1-2481b7754ae5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "87d08375-dbed-4a70-83e1-2481b7754ae5" (UID: "87d08375-dbed-4a70-83e1-2481b7754ae5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 07:01:18.114996 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:18.114966 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c6866d55f-7zvdq" Sep 05 07:01:18.158508 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:18.158480 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87d08375-dbed-4a70-83e1-2481b7754ae5-metrics-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 07:01:18.158508 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:18.158506 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rm5f\" (UniqueName: \"kubernetes.io/projected/87d08375-dbed-4a70-83e1-2481b7754ae5-kube-api-access-8rm5f\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 07:01:18.158508 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:18.158515 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87d08375-dbed-4a70-83e1-2481b7754ae5-maas-api-tls\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Sep 05 07:01:18.341446 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:18.341361 2565 generic.go:356] "Generic (PLEG): container finished" podID="87d08375-dbed-4a70-83e1-2481b7754ae5" containerID="4f3b0e1bb317f963fcea45a3c3238d987836f4add463bd75734d62571ebbd8d4" exitCode=0 Sep 05 07:01:18.341446 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:18.341399 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw" event={"ID":"87d08375-dbed-4a70-83e1-2481b7754ae5","Type":"ContainerDied","Data":"4f3b0e1bb317f963fcea45a3c3238d987836f4add463bd75734d62571ebbd8d4"} Sep 05 07:01:18.341446 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:18.341404 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw" Sep 05 07:01:18.341446 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:18.341446 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw" event={"ID":"87d08375-dbed-4a70-83e1-2481b7754ae5","Type":"ContainerDied","Data":"81796685e830103f2c1c52112cc5810fdb477453bf0f4ca00a6965369906f912"} Sep 05 07:01:18.341920 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:18.341469 2565 scope.go:117] "RemoveContainer" containerID="4f3b0e1bb317f963fcea45a3c3238d987836f4add463bd75734d62571ebbd8d4" Sep 05 07:01:18.350166 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:18.350146 2565 scope.go:117] "RemoveContainer" containerID="4f3b0e1bb317f963fcea45a3c3238d987836f4add463bd75734d62571ebbd8d4" Sep 05 07:01:18.350473 ip-10-0-141-66 kubenswrapper[2565]: E0905 07:01:18.350455 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f3b0e1bb317f963fcea45a3c3238d987836f4add463bd75734d62571ebbd8d4\": container with ID starting with 4f3b0e1bb317f963fcea45a3c3238d987836f4add463bd75734d62571ebbd8d4 not found: ID does not exist" containerID="4f3b0e1bb317f963fcea45a3c3238d987836f4add463bd75734d62571ebbd8d4" Sep 05 07:01:18.350530 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:18.350484 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f3b0e1bb317f963fcea45a3c3238d987836f4add463bd75734d62571ebbd8d4"} err="failed to get container status \"4f3b0e1bb317f963fcea45a3c3238d987836f4add463bd75734d62571ebbd8d4\": rpc error: code = NotFound desc = could not find container \"4f3b0e1bb317f963fcea45a3c3238d987836f4add463bd75734d62571ebbd8d4\": container with ID starting with 4f3b0e1bb317f963fcea45a3c3238d987836f4add463bd75734d62571ebbd8d4 not found: ID does not exist" Sep 05 07:01:18.361736 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:18.359522 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw"] Sep 05 07:01:18.365850 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:18.365820 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-8457d65c99-9vrbw"] Sep 05 07:01:18.391629 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:18.391608 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c6866d55f-7zvdq"] Sep 05 07:01:18.394009 ip-10-0-141-66 kubenswrapper[2565]: W0905 07:01:18.393983 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a022621_eb76_46ed_822f_ac4f70fd4b95.slice/crio-61ce2d092cf91714583f269b595d33613d874f7803e728cfd506932deb2bb5ab WatchSource:0}: Error finding container 61ce2d092cf91714583f269b595d33613d874f7803e728cfd506932deb2bb5ab: Status 404 returned error can't find the container with id 61ce2d092cf91714583f269b595d33613d874f7803e728cfd506932deb2bb5ab Sep 05 07:01:19.348500 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:19.348414 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c6866d55f-7zvdq" event={"ID":"8a022621-eb76-46ed-822f-ac4f70fd4b95","Type":"ContainerStarted","Data":"ccc2e7b90c34932d555e69efd57197ca6864be2dc41ee48050b2b229e34a365a"} Sep 05 07:01:19.348500 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:19.348465 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c6866d55f-7zvdq" event={"ID":"8a022621-eb76-46ed-822f-ac4f70fd4b95","Type":"ContainerStarted","Data":"61ce2d092cf91714583f269b595d33613d874f7803e728cfd506932deb2bb5ab"} Sep 05 07:01:19.348885 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:19.348692 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-c6866d55f-7zvdq" Sep 05 07:01:19.367626 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:19.367579 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-c6866d55f-7zvdq" podStartSLOduration=1.7917761570000001 podStartE2EDuration="2.367565152s" totalImagesPullingTime="575.788995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:01:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:01:19.365039225 +0000 UTC m=+2103.749582349" watchObservedRunningTime="2026-09-05 07:01:19.367565152 +0000 UTC m=+2103.752108276" Sep 05 07:01:20.165827 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:20.165795 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87d08375-dbed-4a70-83e1-2481b7754ae5" path="/var/lib/kubelet/pods/87d08375-dbed-4a70-83e1-2481b7754ae5/volumes" Sep 05 07:01:25.359697 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:25.359668 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-c6866d55f-7zvdq" Sep 05 07:01:50.865736 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:50.865702 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-dcfx4_3100836f-f0c6-4441-9dec-b4e06c85b0a0/manager/0.log" Sep 05 07:01:50.978305 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:50.978274 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-555dc54ddd-f7khx_d2803fb6-d850-4d46-9a7a-7b59edd9326c/manager/0.log" Sep 05 07:01:50.985850 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:50.985826 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-555dc54ddd-f7khx_d2803fb6-d850-4d46-9a7a-7b59edd9326c/copy-manifests/0.log" Sep 05 07:01:51.601300 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:51.601274 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-c6866d55f-7zvdq_8a022621-eb76-46ed-822f-ac4f70fd4b95/maas-api/0.log" Sep 05 07:01:51.922090 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:51.922015 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29809860-pbf97_941e07e0-fb08-4f2b-93f1-ce5f0c41bb27/cleanup/0.log" Sep 05 07:01:53.840960 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:53.840932 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799chzb4t_632ec75c-4f06-4988-bb54-13e262319a5a/istio-proxy/0.log" Sep 05 07:01:53.966816 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:53.966787 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zrvwr_eea75e7c-6676-450b-9796-b531bf8d4120/discovery/0.log" Sep 05 07:01:54.334657 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:54.334629 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-vc22q_405f2706-fa69-4ca4-a14e-44897263f5bb/istio-proxy/1.log" Sep 05 07:01:54.463170 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:54.463134 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-2wvb8_a45a9276-9d3f-402d-9b68-4ffb4462be5b/payload-pre-processing/0.log" Sep 05 07:01:55.038006 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:01:55.037975 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-q86ms_6bfba1c2-a5ae-4fd7-bc6d-87d2808a1c48/main/0.log" Sep 05 07:02:04.709740 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:04.709711 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qzpdk_8a3094f3-d739-47e4-b8a0-49335c888b12/global-pull-secret-syncer/0.log" Sep 05 07:02:04.861426 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:04.861392 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zqj4q_aee7de64-f537-45d5-b9f9-202d8ebc279c/konnectivity-agent/0.log" Sep 05 07:02:04.929431 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:04.929400 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-66.ec2.internal_6a158339afd761bc08604ebeb5aa4b2a/haproxy/0.log" Sep 05 07:02:11.883826 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:11.883802 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-zwhpb_5ac12b19-6ff2-41f9-a7d7-e4416e7962a6/kube-state-metrics/0.log" Sep 05 07:02:11.913796 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:11.913770 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-zwhpb_5ac12b19-6ff2-41f9-a7d7-e4416e7962a6/kube-rbac-proxy-main/0.log" Sep 05 07:02:11.961688 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:11.961662 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-zwhpb_5ac12b19-6ff2-41f9-a7d7-e4416e7962a6/kube-rbac-proxy-self/0.log" Sep 05 07:02:12.003341 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:12.003301 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7c4df55d4f-tw9lb_bceafc69-35f3-4e13-9b38-0024cbfa8899/metrics-server/0.log" Sep 05 07:02:12.050069 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:12.050044 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-tlf9c_0d6de61d-1d35-4d43-b766-a3419dc05d81/monitoring-plugin/0.log" Sep 05 07:02:12.252954 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:12.252880 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v57dr_8cb7ae97-6473-478e-b8e9-534be9f60b02/node-exporter/0.log" Sep 05 07:02:12.278206 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:12.278179 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v57dr_8cb7ae97-6473-478e-b8e9-534be9f60b02/kube-rbac-proxy/0.log" Sep 05 07:02:12.302211 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:12.302186 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v57dr_8cb7ae97-6473-478e-b8e9-534be9f60b02/init-textfile/0.log" Sep 05 07:02:12.332731 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:12.332707 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-pxvmp_03b45c53-3ea5-4817-b164-413cf5f2ae5d/kube-rbac-proxy-main/0.log" Sep 05 07:02:12.357531 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:12.357503 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-pxvmp_03b45c53-3ea5-4817-b164-413cf5f2ae5d/kube-rbac-proxy-self/0.log" Sep 05 07:02:12.380824 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:12.380798 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-pxvmp_03b45c53-3ea5-4817-b164-413cf5f2ae5d/openshift-state-metrics/0.log" Sep 05 07:02:12.793351 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:12.793326 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dcb5fb596-rsdbn_9590c37f-3669-4237-8ac8-78b14e17ce5b/thanos-query/0.log" Sep 05 07:02:12.817263 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:12.817235 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dcb5fb596-rsdbn_9590c37f-3669-4237-8ac8-78b14e17ce5b/kube-rbac-proxy-web/0.log" Sep 05 07:02:12.842363 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:12.842328 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dcb5fb596-rsdbn_9590c37f-3669-4237-8ac8-78b14e17ce5b/kube-rbac-proxy/0.log" Sep 05 07:02:12.874063 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:12.874036 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dcb5fb596-rsdbn_9590c37f-3669-4237-8ac8-78b14e17ce5b/prom-label-proxy/0.log" Sep 05 07:02:12.897055 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:12.897030 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dcb5fb596-rsdbn_9590c37f-3669-4237-8ac8-78b14e17ce5b/kube-rbac-proxy-rules/0.log" Sep 05 07:02:12.926806 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:12.926783 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dcb5fb596-rsdbn_9590c37f-3669-4237-8ac8-78b14e17ce5b/kube-rbac-proxy-metrics/0.log" Sep 05 07:02:14.150633 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.150604 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-92gjz_1185aa3e-9993-48a2-8107-c13c92815d36/networking-console-plugin/0.log" Sep 05 07:02:14.430455 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.430377 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tj5rw/perf-node-gather-daemonset-h5h7r"] Sep 05 07:02:14.430782 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.430769 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87d08375-dbed-4a70-83e1-2481b7754ae5" containerName="maas-api" Sep 05 07:02:14.430831 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.430788 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="87d08375-dbed-4a70-83e1-2481b7754ae5" containerName="maas-api" Sep 05 07:02:14.430866 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.430856 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="87d08375-dbed-4a70-83e1-2481b7754ae5" containerName="maas-api" Sep 05 07:02:14.434226 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.434211 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tj5rw/perf-node-gather-daemonset-h5h7r" Sep 05 07:02:14.436991 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.436966 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tj5rw\"/\"openshift-service-ca.crt\"" Sep 05 07:02:14.437878 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.437858 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tj5rw\"/\"kube-root-ca.crt\"" Sep 05 07:02:14.438011 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.437862 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tj5rw\"/\"default-dockercfg-9d82n\"" Sep 05 07:02:14.442634 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.441891 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tj5rw/perf-node-gather-daemonset-h5h7r"] Sep 05 07:02:14.547192 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.547166 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bee9885d-3a46-46d6-9c5e-096c951eefbd-sys\") pod \"perf-node-gather-daemonset-h5h7r\" (UID: \"bee9885d-3a46-46d6-9c5e-096c951eefbd\") " pod="openshift-must-gather-tj5rw/perf-node-gather-daemonset-h5h7r" Sep 05 07:02:14.547399 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.547208 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bee9885d-3a46-46d6-9c5e-096c951eefbd-lib-modules\") pod \"perf-node-gather-daemonset-h5h7r\" (UID: \"bee9885d-3a46-46d6-9c5e-096c951eefbd\") " pod="openshift-must-gather-tj5rw/perf-node-gather-daemonset-h5h7r" Sep 05 07:02:14.547399 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.547232 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bee9885d-3a46-46d6-9c5e-096c951eefbd-proc\") pod \"perf-node-gather-daemonset-h5h7r\" (UID: \"bee9885d-3a46-46d6-9c5e-096c951eefbd\") " pod="openshift-must-gather-tj5rw/perf-node-gather-daemonset-h5h7r" Sep 05 07:02:14.547399 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.547254 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb5xk\" (UniqueName: \"kubernetes.io/projected/bee9885d-3a46-46d6-9c5e-096c951eefbd-kube-api-access-vb5xk\") pod \"perf-node-gather-daemonset-h5h7r\" (UID: \"bee9885d-3a46-46d6-9c5e-096c951eefbd\") " pod="openshift-must-gather-tj5rw/perf-node-gather-daemonset-h5h7r" Sep 05 07:02:14.547399 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.547351 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bee9885d-3a46-46d6-9c5e-096c951eefbd-podres\") pod \"perf-node-gather-daemonset-h5h7r\" (UID: \"bee9885d-3a46-46d6-9c5e-096c951eefbd\") " pod="openshift-must-gather-tj5rw/perf-node-gather-daemonset-h5h7r" Sep 05 07:02:14.648412 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.648380 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bee9885d-3a46-46d6-9c5e-096c951eefbd-sys\") pod \"perf-node-gather-daemonset-h5h7r\" (UID: \"bee9885d-3a46-46d6-9c5e-096c951eefbd\") " pod="openshift-must-gather-tj5rw/perf-node-gather-daemonset-h5h7r" Sep 05 07:02:14.648583 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.648426 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bee9885d-3a46-46d6-9c5e-096c951eefbd-lib-modules\") pod \"perf-node-gather-daemonset-h5h7r\" (UID: \"bee9885d-3a46-46d6-9c5e-096c951eefbd\") " pod="openshift-must-gather-tj5rw/perf-node-gather-daemonset-h5h7r" Sep 05 07:02:14.648583 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.648448 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bee9885d-3a46-46d6-9c5e-096c951eefbd-proc\") pod \"perf-node-gather-daemonset-h5h7r\" (UID: \"bee9885d-3a46-46d6-9c5e-096c951eefbd\") " pod="openshift-must-gather-tj5rw/perf-node-gather-daemonset-h5h7r" Sep 05 07:02:14.648583 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.648463 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb5xk\" (UniqueName: \"kubernetes.io/projected/bee9885d-3a46-46d6-9c5e-096c951eefbd-kube-api-access-vb5xk\") pod \"perf-node-gather-daemonset-h5h7r\" (UID: \"bee9885d-3a46-46d6-9c5e-096c951eefbd\") " pod="openshift-must-gather-tj5rw/perf-node-gather-daemonset-h5h7r" Sep 05 07:02:14.648583 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.648512 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bee9885d-3a46-46d6-9c5e-096c951eefbd-podres\") pod \"perf-node-gather-daemonset-h5h7r\" (UID: \"bee9885d-3a46-46d6-9c5e-096c951eefbd\") " pod="openshift-must-gather-tj5rw/perf-node-gather-daemonset-h5h7r" Sep 05 07:02:14.648583 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.648517 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bee9885d-3a46-46d6-9c5e-096c951eefbd-sys\") pod \"perf-node-gather-daemonset-h5h7r\" (UID: \"bee9885d-3a46-46d6-9c5e-096c951eefbd\") " pod="openshift-must-gather-tj5rw/perf-node-gather-daemonset-h5h7r" Sep 05 07:02:14.648583 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.648532 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bee9885d-3a46-46d6-9c5e-096c951eefbd-proc\") pod \"perf-node-gather-daemonset-h5h7r\" (UID: \"bee9885d-3a46-46d6-9c5e-096c951eefbd\") " pod="openshift-must-gather-tj5rw/perf-node-gather-daemonset-h5h7r" Sep 05 07:02:14.648583 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.648561 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bee9885d-3a46-46d6-9c5e-096c951eefbd-lib-modules\") pod \"perf-node-gather-daemonset-h5h7r\" (UID: \"bee9885d-3a46-46d6-9c5e-096c951eefbd\") " pod="openshift-must-gather-tj5rw/perf-node-gather-daemonset-h5h7r" Sep 05 07:02:14.648868 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.648622 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bee9885d-3a46-46d6-9c5e-096c951eefbd-podres\") pod \"perf-node-gather-daemonset-h5h7r\" (UID: \"bee9885d-3a46-46d6-9c5e-096c951eefbd\") " pod="openshift-must-gather-tj5rw/perf-node-gather-daemonset-h5h7r" Sep 05 07:02:14.656398 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.656378 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb5xk\" (UniqueName: \"kubernetes.io/projected/bee9885d-3a46-46d6-9c5e-096c951eefbd-kube-api-access-vb5xk\") pod \"perf-node-gather-daemonset-h5h7r\" (UID: \"bee9885d-3a46-46d6-9c5e-096c951eefbd\") " pod="openshift-must-gather-tj5rw/perf-node-gather-daemonset-h5h7r" Sep 05 07:02:14.746097 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:14.746033 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tj5rw/perf-node-gather-daemonset-h5h7r" Sep 05 07:02:15.015878 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:15.015845 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tj5rw/perf-node-gather-daemonset-h5h7r"] Sep 05 07:02:15.018247 ip-10-0-141-66 kubenswrapper[2565]: W0905 07:02:15.018215 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbee9885d_3a46_46d6_9c5e_096c951eefbd.slice/crio-fd719feaa75e9966a3e3c9408ed28cbfdaf642a9db30e9f6293453b288c5f162 WatchSource:0}: Error finding container fd719feaa75e9966a3e3c9408ed28cbfdaf642a9db30e9f6293453b288c5f162: Status 404 returned error can't find the container with id fd719feaa75e9966a3e3c9408ed28cbfdaf642a9db30e9f6293453b288c5f162 Sep 05 07:02:15.620019 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:15.619987 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tj5rw/perf-node-gather-daemonset-h5h7r" event={"ID":"bee9885d-3a46-46d6-9c5e-096c951eefbd","Type":"ContainerStarted","Data":"719995a5b15b5dd569aec24b3d0dce374848c5fc26242da7fdbc1878edb8f57a"} Sep 05 07:02:15.620417 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:15.620034 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tj5rw/perf-node-gather-daemonset-h5h7r" event={"ID":"bee9885d-3a46-46d6-9c5e-096c951eefbd","Type":"ContainerStarted","Data":"fd719feaa75e9966a3e3c9408ed28cbfdaf642a9db30e9f6293453b288c5f162"} Sep 05 07:02:15.620417 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:15.620060 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tj5rw/perf-node-gather-daemonset-h5h7r" Sep 05 07:02:15.635056 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:15.635011 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tj5rw/perf-node-gather-daemonset-h5h7r" podStartSLOduration=1.6349921539999999 podStartE2EDuration="1.634992154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 07:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:02:15.634917913 +0000 UTC m=+2160.019461038" watchObservedRunningTime="2026-09-05 07:02:15.634992154 +0000 UTC m=+2160.019535281" Sep 05 07:02:16.624117 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:16.624087 2565 scope.go:117] "RemoveContainer" containerID="1a7212f7a1f62ad5b8dbdc72138eb9fcec02e1294bc5e04b90d09021a8264cb1" Sep 05 07:02:16.632218 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:16.632199 2565 scope.go:117] "RemoveContainer" containerID="c31caccbc61e34f5dd71c727374bdde83821cbfa2e298cfeba1813339212f959" Sep 05 07:02:17.149190 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:17.149167 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j4dq5_3f8f4891-a720-43c1-bdf3-73b8250fde4f/dns/0.log" Sep 05 07:02:17.172535 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:17.172512 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j4dq5_3f8f4891-a720-43c1-bdf3-73b8250fde4f/kube-rbac-proxy/0.log" Sep 05 07:02:17.298325 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:17.298280 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dzlmk_59539926-4278-4795-8ff8-f3482b69df83/dns-node-resolver/0.log" Sep 05 07:02:17.891131 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:17.891098 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7759f7f78b-bvxqx_70b7f437-85ac-49f8-bb86-1266ac5db0b6/registry/0.log" Sep 05 07:02:17.969069 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:17.969040 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m7rvs_60b02f38-b310-4273-bc86-e3eb032c1cd4/node-ca/0.log" Sep 05 07:02:18.898849 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:18.898808 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799chzb4t_632ec75c-4f06-4988-bb54-13e262319a5a/istio-proxy/0.log" Sep 05 07:02:19.189673 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:19.189595 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zrvwr_eea75e7c-6676-450b-9796-b531bf8d4120/discovery/0.log" Sep 05 07:02:19.270231 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:19.270198 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-vc22q_405f2706-fa69-4ca4-a14e-44897263f5bb/istio-proxy/0.log" Sep 05 07:02:19.279063 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:19.279043 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-vc22q_405f2706-fa69-4ca4-a14e-44897263f5bb/istio-proxy/1.log" Sep 05 07:02:19.312599 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:19.312581 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-2wvb8_a45a9276-9d3f-402d-9b68-4ffb4462be5b/payload-pre-processing/0.log" Sep 05 07:02:20.260004 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:20.259974 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cmxtg_ffad8c9d-5409-4c11-a250-686bb69099d4/serve-healthcheck-canary/0.log" Sep 05 07:02:20.790627 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:20.790602 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-97l2v_b2ecc993-91c0-426b-94e1-664c820c0c57/kube-rbac-proxy/0.log" Sep 05 07:02:20.811742 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:20.811720 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-97l2v_b2ecc993-91c0-426b-94e1-664c820c0c57/exporter/0.log" Sep 05 07:02:20.832649 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:20.832624 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-97l2v_b2ecc993-91c0-426b-94e1-664c820c0c57/extractor/0.log" Sep 05 07:02:21.636658 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:21.636634 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tj5rw/perf-node-gather-daemonset-h5h7r" Sep 05 07:02:23.584053 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:23.583980 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-dcfx4_3100836f-f0c6-4441-9dec-b4e06c85b0a0/manager/0.log" Sep 05 07:02:23.610346 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:23.610300 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-555dc54ddd-f7khx_d2803fb6-d850-4d46-9a7a-7b59edd9326c/manager/0.log" Sep 05 07:02:23.633053 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:23.633034 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-555dc54ddd-f7khx_d2803fb6-d850-4d46-9a7a-7b59edd9326c/copy-manifests/0.log" Sep 05 07:02:24.942057 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:24.942024 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799chzb4t_632ec75c-4f06-4988-bb54-13e262319a5a/istio-proxy/0.log" Sep 05 07:02:25.196701 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:25.196626 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-66f494d4f4-vnfk5_4ac01a53-7ea0-4d67-8fb6-6e918f6c06e4/manager/0.log" Sep 05 07:02:25.226876 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:25.226850 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-zbckx_1bc7cfc7-cf93-499e-a3fb-22c1b28fb662/openshift-lws-operator/0.log" Sep 05 07:02:25.238051 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:25.238028 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zrvwr_eea75e7c-6676-450b-9796-b531bf8d4120/discovery/0.log" Sep 05 07:02:25.290854 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:25.290829 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-vc22q_405f2706-fa69-4ca4-a14e-44897263f5bb/istio-proxy/0.log" Sep 05 07:02:25.301700 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:25.301678 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-vc22q_405f2706-fa69-4ca4-a14e-44897263f5bb/istio-proxy/1.log" Sep 05 07:02:25.328826 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:25.328803 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-2wvb8_a45a9276-9d3f-402d-9b68-4ffb4462be5b/payload-pre-processing/0.log" Sep 05 07:02:26.816222 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:26.816195 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799chzb4t_632ec75c-4f06-4988-bb54-13e262319a5a/istio-proxy/0.log" Sep 05 07:02:27.098759 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:27.098700 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zrvwr_eea75e7c-6676-450b-9796-b531bf8d4120/discovery/0.log" Sep 05 07:02:27.159691 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:27.159661 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-vc22q_405f2706-fa69-4ca4-a14e-44897263f5bb/istio-proxy/0.log" Sep 05 07:02:27.169295 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:27.169275 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-vc22q_405f2706-fa69-4ca4-a14e-44897263f5bb/istio-proxy/1.log" Sep 05 07:02:27.194619 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:27.194597 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-2wvb8_a45a9276-9d3f-402d-9b68-4ffb4462be5b/payload-pre-processing/0.log" Sep 05 07:02:30.671592 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:30.671563 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-k68n5_62636c45-1709-4102-bbfc-a13e5f699d1e/kube-storage-version-migrator-operator/1.log" Sep 05 07:02:30.672233 ip-10-0-141-66 kubenswrapper[2565]: I0905 07:02:30.672217 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-k68n5_62636c45-1709-4102-bbfc-a13e5f699d1e/kube-storage-version-migrator-operator/0.log"