Sep 14 10:58:56.631963 ip-10-0-143-52 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 14 10:58:56.631975 ip-10-0-143-52 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 14 10:58:56.631983 ip-10-0-143-52 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 14 10:58:56.632176 ip-10-0-143-52 systemd[1]: Failed to start Kubernetes Kubelet. Sep 14 10:59:06.665637 ip-10-0-143-52 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 14 10:59:06.665655 ip-10-0-143-52 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 397c11240f6a4ef59d90d3874edd377b -- Sep 14 11:00:41.817405 ip-10-0-143-52 systemd[1]: Starting Kubernetes Kubelet... Sep 14 11:00:42.196356 ip-10-0-143-52 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 11:00:42.196356 ip-10-0-143-52 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 14 11:00:42.196356 ip-10-0-143-52 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 11:00:42.196356 ip-10-0-143-52 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 14 11:00:42.196356 ip-10-0-143-52 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 11:00:42.197236 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.197142 2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 14 11:00:42.200015 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.199999 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 11:00:42.200015 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200016 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 11:00:42.200076 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200019 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 11:00:42.200076 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200023 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 11:00:42.200076 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200026 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 11:00:42.200076 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200029 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 11:00:42.200076 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200032 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 11:00:42.200076 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200034 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 11:00:42.200076 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200037 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 11:00:42.200076 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200040 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 11:00:42.200076 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200043 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 11:00:42.200076 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200046 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 11:00:42.200076 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200049 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 11:00:42.200076 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200052 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 11:00:42.200076 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200054 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 11:00:42.200076 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200062 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 11:00:42.200076 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200065 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 11:00:42.200076 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200068 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 11:00:42.200076 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200070 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 11:00:42.200076 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200073 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 11:00:42.200076 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200076 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 11:00:42.200076 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200079 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 11:00:42.200569 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200081 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 11:00:42.200569 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200096 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 11:00:42.200569 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200099 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 11:00:42.200569 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200102 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 11:00:42.200569 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200105 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 11:00:42.200569 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200107 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 11:00:42.200569 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200110 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 11:00:42.200569 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200113 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 11:00:42.200569 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200116 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 11:00:42.200569 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200118 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 11:00:42.200569 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200121 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 11:00:42.200569 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200124 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 11:00:42.200569 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200127 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 11:00:42.200569 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200129 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 11:00:42.200569 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200134 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 11:00:42.200569 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200138 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 11:00:42.200569 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200140 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 11:00:42.200569 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200143 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 11:00:42.200569 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200146 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 11:00:42.201071 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200148 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 11:00:42.201071 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200151 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 11:00:42.201071 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200154 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 11:00:42.201071 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200157 2556 feature_gate.go:328] unrecognized feature gate: Example Sep 14 11:00:42.201071 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200159 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 11:00:42.201071 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200162 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 11:00:42.201071 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200164 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 11:00:42.201071 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200167 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 11:00:42.201071 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200169 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 11:00:42.201071 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200172 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 11:00:42.201071 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200174 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 11:00:42.201071 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200176 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 11:00:42.201071 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200179 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 11:00:42.201071 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200182 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 11:00:42.201071 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200186 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 11:00:42.201071 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200189 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 11:00:42.201071 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200191 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 11:00:42.201071 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200194 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 11:00:42.201071 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200196 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 11:00:42.201560 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200199 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 11:00:42.201560 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200202 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 11:00:42.201560 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200204 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 11:00:42.201560 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200207 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 11:00:42.201560 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200210 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 11:00:42.201560 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200212 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 11:00:42.201560 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200215 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 11:00:42.201560 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200217 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 11:00:42.201560 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200220 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 11:00:42.201560 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200222 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 11:00:42.201560 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200224 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 11:00:42.201560 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200229 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 11:00:42.201560 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200232 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 11:00:42.201560 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200235 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 11:00:42.201560 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200238 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 11:00:42.201560 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200242 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 11:00:42.201560 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200245 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 11:00:42.201560 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200249 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 11:00:42.201560 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200252 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 11:00:42.202022 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200254 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 11:00:42.202022 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200257 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 11:00:42.202022 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200260 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 11:00:42.202022 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200263 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 11:00:42.202022 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200266 2556 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 11:00:42.202022 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200269 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 11:00:42.202022 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200271 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 11:00:42.202022 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200274 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 11:00:42.202022 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200279 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 11:00:42.202022 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200281 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 11:00:42.202022 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200284 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 11:00:42.202022 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200287 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 11:00:42.202022 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200289 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 11:00:42.202022 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200292 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 11:00:42.202022 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200294 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 11:00:42.202022 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200297 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 11:00:42.202022 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200299 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 11:00:42.202022 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200302 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 11:00:42.202022 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200305 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 11:00:42.202022 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.200307 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 11:00:42.202514 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201024 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 11:00:42.202514 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201032 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 11:00:42.202514 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201036 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 11:00:42.202514 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201038 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 11:00:42.202514 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201042 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 11:00:42.202514 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201045 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 11:00:42.202514 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201047 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 11:00:42.202514 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201050 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 11:00:42.202514 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201053 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 11:00:42.202514 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201056 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 11:00:42.202514 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201058 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 11:00:42.202514 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201062 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 11:00:42.202514 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201065 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 11:00:42.202514 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201070 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 11:00:42.202514 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201073 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 11:00:42.202514 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201076 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 11:00:42.202514 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201079 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 11:00:42.202514 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201081 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 11:00:42.202514 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201098 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 11:00:42.202514 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201101 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 11:00:42.203002 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201104 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 11:00:42.203002 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201107 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 11:00:42.203002 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201109 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 11:00:42.203002 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201112 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 11:00:42.203002 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201114 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 11:00:42.203002 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201117 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 11:00:42.203002 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201120 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 11:00:42.203002 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201122 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 11:00:42.203002 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201125 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 11:00:42.203002 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201128 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 11:00:42.203002 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201130 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 11:00:42.203002 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201133 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 11:00:42.203002 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201135 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 11:00:42.203002 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201138 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 11:00:42.203002 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201140 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 11:00:42.203002 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201143 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 11:00:42.203002 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201145 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 11:00:42.203002 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201147 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 11:00:42.203002 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201150 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 11:00:42.203499 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201155 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 11:00:42.203499 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201158 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 11:00:42.203499 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201162 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 11:00:42.203499 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201165 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 11:00:42.203499 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201169 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 11:00:42.203499 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201179 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 11:00:42.203499 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201182 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 11:00:42.203499 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201185 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 11:00:42.203499 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201188 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 11:00:42.203499 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201191 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 11:00:42.203499 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201193 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 11:00:42.203499 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201196 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 11:00:42.203499 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201199 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 11:00:42.203499 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201202 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 11:00:42.203499 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201204 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 11:00:42.203499 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201206 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 11:00:42.203499 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201209 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 11:00:42.203499 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201212 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 11:00:42.203938 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201214 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 11:00:42.203938 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201216 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 11:00:42.203938 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201219 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 11:00:42.203938 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201221 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 11:00:42.203938 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201224 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 11:00:42.203938 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201226 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 11:00:42.203938 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201229 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 11:00:42.203938 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201231 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 11:00:42.203938 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201234 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 11:00:42.203938 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201236 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 11:00:42.203938 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201238 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 11:00:42.203938 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201241 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 11:00:42.203938 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201243 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 11:00:42.203938 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201247 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 11:00:42.203938 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201249 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 11:00:42.203938 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201253 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 11:00:42.203938 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201256 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 11:00:42.203938 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201258 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 11:00:42.203938 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201261 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 11:00:42.203938 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201265 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 11:00:42.204448 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201268 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 11:00:42.204448 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201270 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 11:00:42.204448 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201273 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 11:00:42.204448 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201275 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 11:00:42.204448 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201278 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 11:00:42.204448 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201280 2556 feature_gate.go:328] unrecognized feature gate: Example Sep 14 11:00:42.204448 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201283 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 11:00:42.204448 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201285 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 11:00:42.204448 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201288 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 11:00:42.204448 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201290 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 11:00:42.204448 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201293 2556 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 11:00:42.204448 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201295 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 11:00:42.204448 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201298 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 11:00:42.204448 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201300 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 11:00:42.204448 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201303 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 11:00:42.204448 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201305 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 11:00:42.204448 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201308 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 11:00:42.204448 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201310 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 11:00:42.204448 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201313 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 11:00:42.204448 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201315 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 11:00:42.204448 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201317 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 11:00:42.204965 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201320 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 11:00:42.204965 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201678 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 11:00:42.204965 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201683 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 11:00:42.204965 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201686 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 11:00:42.204965 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201689 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 11:00:42.204965 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201691 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 11:00:42.204965 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201694 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 11:00:42.204965 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201698 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 11:00:42.204965 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201701 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 11:00:42.204965 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201704 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 11:00:42.204965 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201708 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 11:00:42.204965 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201711 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 11:00:42.204965 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201714 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 11:00:42.204965 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201717 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 11:00:42.204965 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201719 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 11:00:42.204965 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201722 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 11:00:42.204965 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201724 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 11:00:42.204965 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201727 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 11:00:42.204965 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201730 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 11:00:42.204965 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201732 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 11:00:42.205459 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201735 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 11:00:42.205459 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201737 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 11:00:42.205459 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201740 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 11:00:42.205459 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201742 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 11:00:42.205459 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201745 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 11:00:42.205459 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201747 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 11:00:42.205459 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201750 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 11:00:42.205459 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201753 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 11:00:42.205459 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201757 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 11:00:42.205459 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201760 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 11:00:42.205459 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201763 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 11:00:42.205459 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201766 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 11:00:42.205459 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201769 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 11:00:42.205459 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201771 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 11:00:42.205459 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201776 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 11:00:42.205459 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201778 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 11:00:42.205459 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201782 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 11:00:42.205459 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201784 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 11:00:42.205459 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201787 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 11:00:42.205916 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201790 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 11:00:42.205916 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201792 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 11:00:42.205916 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201797 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 11:00:42.205916 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201800 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 11:00:42.205916 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201803 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 11:00:42.205916 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201806 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 11:00:42.205916 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201809 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 11:00:42.205916 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201812 2556 feature_gate.go:328] unrecognized feature gate: Example Sep 14 11:00:42.205916 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201815 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 11:00:42.205916 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201817 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 11:00:42.205916 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201820 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 11:00:42.205916 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201822 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 11:00:42.205916 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201825 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 11:00:42.205916 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201827 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 11:00:42.205916 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201830 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 11:00:42.205916 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201832 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 11:00:42.205916 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201835 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 11:00:42.205916 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201837 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 11:00:42.205916 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201840 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 11:00:42.205916 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201842 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 11:00:42.206401 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201845 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 11:00:42.206401 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201847 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 11:00:42.206401 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201850 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 11:00:42.206401 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201853 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 11:00:42.206401 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201855 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 11:00:42.206401 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201858 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 11:00:42.206401 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201860 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 11:00:42.206401 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201865 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 11:00:42.206401 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201867 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 11:00:42.206401 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201870 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 11:00:42.206401 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201872 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 11:00:42.206401 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201875 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 11:00:42.206401 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201878 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 11:00:42.206401 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201880 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 11:00:42.206401 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201884 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 11:00:42.206401 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201887 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 11:00:42.206401 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201889 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 11:00:42.206401 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201892 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 11:00:42.206401 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201895 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 11:00:42.206401 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201897 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 11:00:42.206882 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201900 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 11:00:42.206882 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201902 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 11:00:42.206882 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201904 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 11:00:42.206882 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201907 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 11:00:42.206882 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201910 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 11:00:42.206882 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201912 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 11:00:42.206882 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201915 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 11:00:42.206882 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201917 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 11:00:42.206882 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201920 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 11:00:42.206882 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201922 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 11:00:42.206882 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201925 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 11:00:42.206882 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201927 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 11:00:42.206882 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201929 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 11:00:42.206882 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201932 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 11:00:42.206882 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201934 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 11:00:42.206882 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201937 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 11:00:42.206882 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201939 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 11:00:42.206882 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201942 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 11:00:42.206882 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201944 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 11:00:42.206882 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201949 2556 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 11:00:42.207363 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.201951 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 11:00:42.207363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202030 2556 flags.go:64] FLAG: --address="0.0.0.0" Sep 14 11:00:42.207363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202065 2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 14 11:00:42.207363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202072 2556 flags.go:64] FLAG: --anonymous-auth="true" Sep 14 11:00:42.207363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202076 2556 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 14 11:00:42.207363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202080 2556 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 14 11:00:42.207363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202098 2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 14 11:00:42.207363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202103 2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 14 11:00:42.207363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202107 2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 14 11:00:42.207363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202111 2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 14 11:00:42.207363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202114 2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 14 11:00:42.207363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202118 2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 14 11:00:42.207363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202121 2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 14 11:00:42.207363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202125 2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 14 11:00:42.207363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202127 2556 flags.go:64] FLAG: --cgroup-root="" Sep 14 11:00:42.207363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202130 2556 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 14 11:00:42.207363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202134 2556 flags.go:64] FLAG: --client-ca-file="" Sep 14 11:00:42.207363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202137 2556 flags.go:64] FLAG: --cloud-provider="external" Sep 14 11:00:42.207363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202140 2556 flags.go:64] FLAG: --cluster-dns="[]" Sep 14 11:00:42.207363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202145 2556 flags.go:64] FLAG: --cluster-domain="" Sep 14 11:00:42.207363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202148 2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 14 11:00:42.207363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202151 2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 14 11:00:42.207363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202154 2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 14 11:00:42.207363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202157 2556 flags.go:64] FLAG: --container-log-max-files="5" Sep 14 11:00:42.207956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202163 2556 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 14 11:00:42.207956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202166 2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 14 11:00:42.207956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202169 2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 14 11:00:42.207956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202173 2556 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 14 11:00:42.207956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202176 2556 flags.go:64] FLAG: --contention-profiling="false" Sep 14 11:00:42.207956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202179 2556 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 14 11:00:42.207956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202182 2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 14 11:00:42.207956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202185 2556 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 14 11:00:42.207956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202188 2556 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 14 11:00:42.207956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202193 2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 14 11:00:42.207956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202195 2556 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 14 11:00:42.207956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202198 2556 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 14 11:00:42.207956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202202 2556 flags.go:64] FLAG: --enable-load-reader="false" Sep 14 11:00:42.207956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202205 2556 flags.go:64] FLAG: --enable-server="true" Sep 14 11:00:42.207956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202209 2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 14 11:00:42.207956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202213 2556 flags.go:64] FLAG: --event-burst="100" Sep 14 11:00:42.207956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202216 2556 flags.go:64] FLAG: --event-qps="50" Sep 14 11:00:42.207956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202219 2556 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 14 11:00:42.207956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202222 2556 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 14 11:00:42.207956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202225 2556 flags.go:64] FLAG: --eviction-hard="" Sep 14 11:00:42.207956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202229 2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 14 11:00:42.207956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202232 2556 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 14 11:00:42.207956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202236 2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 14 11:00:42.207956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202238 2556 flags.go:64] FLAG: --eviction-soft="" Sep 14 11:00:42.207956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202241 2556 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 14 11:00:42.208583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202244 2556 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 14 11:00:42.208583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202247 2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 14 11:00:42.208583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202250 2556 flags.go:64] FLAG: --experimental-mounter-path="" Sep 14 11:00:42.208583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202253 2556 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 14 11:00:42.208583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202256 2556 flags.go:64] FLAG: --fail-swap-on="true" Sep 14 11:00:42.208583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202258 2556 flags.go:64] FLAG: --feature-gates="" Sep 14 11:00:42.208583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202262 2556 flags.go:64] FLAG: --file-check-frequency="20s" Sep 14 11:00:42.208583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202265 2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 14 11:00:42.208583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202268 2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 14 11:00:42.208583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202271 2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 14 11:00:42.208583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202273 2556 flags.go:64] FLAG: --healthz-port="10248" Sep 14 11:00:42.208583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202276 2556 flags.go:64] FLAG: --help="false" Sep 14 11:00:42.208583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202279 2556 flags.go:64] FLAG: --hostname-override="ip-10-0-143-52.ec2.internal" Sep 14 11:00:42.208583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202282 2556 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 14 11:00:42.208583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202285 2556 flags.go:64] FLAG: --http-check-frequency="20s" Sep 14 11:00:42.208583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202288 2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 14 11:00:42.208583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202292 2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 14 11:00:42.208583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202295 2556 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 14 11:00:42.208583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202298 2556 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 14 11:00:42.208583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202301 2556 flags.go:64] FLAG: --image-service-endpoint="" Sep 14 11:00:42.208583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202304 2556 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 14 11:00:42.208583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202308 2556 flags.go:64] FLAG: --kube-api-burst="100" Sep 14 11:00:42.208583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202310 2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 14 11:00:42.209667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202314 2556 flags.go:64] FLAG: --kube-api-qps="50" Sep 14 11:00:42.209667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202316 2556 flags.go:64] FLAG: --kube-reserved="" Sep 14 11:00:42.209667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202319 2556 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 14 11:00:42.209667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202322 2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 14 11:00:42.209667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202325 2556 flags.go:64] FLAG: --kubelet-cgroups="" Sep 14 11:00:42.209667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202328 2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 14 11:00:42.209667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202331 2556 flags.go:64] FLAG: --lock-file="" Sep 14 11:00:42.209667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202334 2556 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 14 11:00:42.209667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202336 2556 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 14 11:00:42.209667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202339 2556 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 14 11:00:42.209667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202345 2556 flags.go:64] FLAG: --log-json-split-stream="false" Sep 14 11:00:42.209667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202347 2556 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 14 11:00:42.209667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202350 2556 flags.go:64] FLAG: --log-text-split-stream="false" Sep 14 11:00:42.209667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202353 2556 flags.go:64] FLAG: --logging-format="text" Sep 14 11:00:42.209667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202356 2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 14 11:00:42.209667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202359 2556 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 14 11:00:42.209667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202362 2556 flags.go:64] FLAG: --manifest-url="" Sep 14 11:00:42.209667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202365 2556 flags.go:64] FLAG: --manifest-url-header="" Sep 14 11:00:42.209667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202369 2556 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 14 11:00:42.209667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202372 2556 flags.go:64] FLAG: --max-open-files="1000000" Sep 14 11:00:42.209667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202376 2556 flags.go:64] FLAG: --max-pods="110" Sep 14 11:00:42.209667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202379 2556 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 14 11:00:42.209667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202381 2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 14 11:00:42.209667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202385 2556 flags.go:64] FLAG: --memory-manager-policy="None" Sep 14 11:00:42.209667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202388 2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 14 11:00:42.210774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202391 2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 14 11:00:42.210774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202394 2556 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 14 11:00:42.210774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202397 2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 14 11:00:42.210774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202404 2556 flags.go:64] FLAG: --node-status-max-images="50" Sep 14 11:00:42.210774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202407 2556 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 14 11:00:42.210774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202412 2556 flags.go:64] FLAG: --oom-score-adj="-999" Sep 14 11:00:42.210774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202415 2556 flags.go:64] FLAG: --pod-cidr="" Sep 14 11:00:42.210774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202418 2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 14 11:00:42.210774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202423 2556 flags.go:64] FLAG: --pod-manifest-path="" Sep 14 11:00:42.210774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202426 2556 flags.go:64] FLAG: --pod-max-pids="-1" Sep 14 11:00:42.210774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202429 2556 flags.go:64] FLAG: --pods-per-core="0" Sep 14 11:00:42.210774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202432 2556 flags.go:64] FLAG: --port="10250" Sep 14 11:00:42.210774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202435 2556 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 14 11:00:42.210774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202439 2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0de1ba6f91036491d" Sep 14 11:00:42.210774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202442 2556 flags.go:64] FLAG: --qos-reserved="" Sep 14 11:00:42.210774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202445 2556 flags.go:64] FLAG: --read-only-port="10255" Sep 14 11:00:42.210774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202448 2556 flags.go:64] FLAG: --register-node="true" Sep 14 11:00:42.210774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202451 2556 flags.go:64] FLAG: --register-with-taints="" Sep 14 11:00:42.210774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202454 2556 flags.go:64] FLAG: --registry-burst="10" Sep 14 11:00:42.210774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202457 2556 flags.go:64] FLAG: --registry-qps="5" Sep 14 11:00:42.210774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202460 2556 flags.go:64] FLAG: --reserved-cpus="" Sep 14 11:00:42.210774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202463 2556 flags.go:64] FLAG: --reserved-memory="" Sep 14 11:00:42.210774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202467 2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 14 11:00:42.210774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202470 2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 14 11:00:42.210774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202473 2556 flags.go:64] FLAG: --rotate-certificates="false" Sep 14 11:00:42.211674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202476 2556 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 14 11:00:42.211674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202479 2556 flags.go:64] FLAG: --runonce="false" Sep 14 11:00:42.211674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202482 2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 14 11:00:42.211674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202485 2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 14 11:00:42.211674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202488 2556 flags.go:64] FLAG: --seccomp-default="false" Sep 14 11:00:42.211674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202491 2556 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 14 11:00:42.211674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202494 2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 14 11:00:42.211674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202497 2556 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 14 11:00:42.211674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202500 2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 14 11:00:42.211674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202503 2556 flags.go:64] FLAG: --storage-driver-password="root" Sep 14 11:00:42.211674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202506 2556 flags.go:64] FLAG: --storage-driver-secure="false" Sep 14 11:00:42.211674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202509 2556 flags.go:64] FLAG: --storage-driver-table="stats" Sep 14 11:00:42.211674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202513 2556 flags.go:64] FLAG: --storage-driver-user="root" Sep 14 11:00:42.211674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202516 2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 14 11:00:42.211674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202519 2556 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 14 11:00:42.211674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202522 2556 flags.go:64] FLAG: --system-cgroups="" Sep 14 11:00:42.211674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202525 2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 14 11:00:42.211674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202531 2556 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 14 11:00:42.211674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202534 2556 flags.go:64] FLAG: --tls-cert-file="" Sep 14 11:00:42.211674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202537 2556 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 14 11:00:42.211674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202541 2556 flags.go:64] FLAG: --tls-min-version="" Sep 14 11:00:42.211674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202544 2556 flags.go:64] FLAG: --tls-private-key-file="" Sep 14 11:00:42.211674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202547 2556 flags.go:64] FLAG: --topology-manager-policy="none" Sep 14 11:00:42.211674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202550 2556 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 14 11:00:42.211674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202553 2556 flags.go:64] FLAG: --topology-manager-scope="container" Sep 14 11:00:42.212554 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202555 2556 flags.go:64] FLAG: --v="2" Sep 14 11:00:42.212554 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202559 2556 flags.go:64] FLAG: --version="false" Sep 14 11:00:42.212554 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202563 2556 flags.go:64] FLAG: --vmodule="" Sep 14 11:00:42.212554 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202567 2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 14 11:00:42.212554 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202570 2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 14 11:00:42.212554 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202683 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 11:00:42.212554 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202686 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 11:00:42.212554 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202689 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 11:00:42.212554 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202692 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 11:00:42.212554 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202695 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 11:00:42.212554 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202698 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 11:00:42.212554 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202701 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 11:00:42.212554 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202706 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 11:00:42.212554 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202709 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 11:00:42.212554 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202711 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 11:00:42.212554 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202714 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 11:00:42.212554 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202717 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 11:00:42.212554 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202720 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 11:00:42.212554 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202723 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 11:00:42.212554 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202726 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 11:00:42.212554 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202729 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 11:00:42.213492 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202732 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 11:00:42.213492 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202734 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 11:00:42.213492 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202737 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 11:00:42.213492 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202739 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 11:00:42.213492 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202742 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 11:00:42.213492 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202745 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 11:00:42.213492 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202747 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 11:00:42.213492 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202750 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 11:00:42.213492 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202753 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 11:00:42.213492 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202756 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 11:00:42.213492 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202759 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 11:00:42.213492 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202761 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 11:00:42.213492 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202764 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 11:00:42.213492 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202767 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 11:00:42.213492 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202769 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 11:00:42.213492 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202772 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 11:00:42.213492 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202774 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 11:00:42.213492 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202777 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 11:00:42.213492 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202779 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 11:00:42.213492 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202782 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 11:00:42.214105 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202784 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 11:00:42.214105 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202787 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 11:00:42.214105 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202789 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 11:00:42.214105 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202793 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 11:00:42.214105 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202796 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 11:00:42.214105 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202798 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 11:00:42.214105 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202801 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 11:00:42.214105 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202803 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 11:00:42.214105 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202806 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 11:00:42.214105 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202809 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 11:00:42.214105 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202813 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 11:00:42.214105 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202817 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 11:00:42.214105 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202821 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 11:00:42.214105 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202824 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 11:00:42.214105 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202827 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 11:00:42.214105 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202831 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 11:00:42.214105 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202834 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 11:00:42.214105 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202837 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 11:00:42.214105 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202840 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 11:00:42.214854 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202843 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 11:00:42.214854 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202846 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 11:00:42.214854 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202849 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 11:00:42.214854 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202851 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 11:00:42.214854 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202854 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 11:00:42.214854 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202856 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 11:00:42.214854 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202859 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 11:00:42.214854 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202861 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 11:00:42.214854 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202864 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 11:00:42.214854 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202867 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 11:00:42.214854 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202869 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 11:00:42.214854 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202872 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 11:00:42.214854 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202874 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 11:00:42.214854 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202876 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 11:00:42.214854 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202879 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 11:00:42.214854 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202881 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 11:00:42.214854 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202885 2556 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 11:00:42.214854 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202888 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 11:00:42.214854 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202890 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 11:00:42.214854 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202893 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 11:00:42.215408 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202896 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 11:00:42.215408 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202899 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 11:00:42.215408 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202902 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 11:00:42.215408 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202904 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 11:00:42.215408 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202907 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 11:00:42.215408 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202909 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 11:00:42.215408 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202912 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 11:00:42.215408 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202915 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 11:00:42.215408 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202918 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 11:00:42.215408 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202920 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 11:00:42.215408 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202923 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 11:00:42.215408 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202926 2556 feature_gate.go:328] unrecognized feature gate: Example Sep 14 11:00:42.215408 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202928 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 11:00:42.215408 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202931 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 11:00:42.215408 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202933 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 11:00:42.215408 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202936 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 11:00:42.215408 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202939 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 11:00:42.215408 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202941 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 11:00:42.215408 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202943 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 11:00:42.215408 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202946 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 11:00:42.215905 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202948 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 11:00:42.215905 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202951 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 11:00:42.215905 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202953 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 11:00:42.215905 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.202956 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 11:00:42.215905 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.202961 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 11:00:42.215905 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.211234 2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 14 11:00:42.215905 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.211258 2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 14 11:00:42.215905 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.212047 2556 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 14 11:00:42.215905 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.212065 2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 14 11:00:42.215905 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212178 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 11:00:42.215905 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212188 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 11:00:42.215905 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212196 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 11:00:42.215905 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212201 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 11:00:42.215905 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212206 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 11:00:42.216574 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212211 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 11:00:42.216574 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212215 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 11:00:42.216574 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212220 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 11:00:42.216574 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212225 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 11:00:42.216574 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212229 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 11:00:42.216574 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212234 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 11:00:42.216574 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212239 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 11:00:42.216574 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212243 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 11:00:42.216574 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212247 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 11:00:42.216574 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212251 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 11:00:42.216574 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212255 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 11:00:42.216574 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212260 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 11:00:42.216574 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212264 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 11:00:42.216574 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212268 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 11:00:42.216574 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212272 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 11:00:42.216574 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212276 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 11:00:42.216574 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212280 2556 feature_gate.go:328] unrecognized feature gate: Example Sep 14 11:00:42.216574 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212285 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 11:00:42.216574 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212289 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 11:00:42.216574 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212293 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 11:00:42.217416 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212297 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 11:00:42.217416 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212301 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 11:00:42.217416 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212305 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 11:00:42.217416 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212309 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 11:00:42.217416 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212323 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 11:00:42.217416 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212328 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 11:00:42.217416 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212332 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 11:00:42.217416 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212336 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 11:00:42.217416 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212340 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 11:00:42.217416 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212344 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 11:00:42.217416 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212349 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 11:00:42.217416 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212353 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 11:00:42.217416 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212357 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 11:00:42.217416 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212362 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 11:00:42.217416 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212366 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 11:00:42.217416 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212370 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 11:00:42.217416 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212374 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 11:00:42.217416 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212378 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 11:00:42.217416 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212383 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 11:00:42.217416 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212387 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 11:00:42.217952 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212391 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 11:00:42.217952 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212395 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 11:00:42.217952 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212399 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 11:00:42.217952 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212403 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 11:00:42.217952 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212407 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 11:00:42.217952 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212412 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 11:00:42.217952 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212416 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 11:00:42.217952 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212420 2556 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 11:00:42.217952 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212424 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 11:00:42.217952 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212428 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 11:00:42.217952 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212432 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 11:00:42.217952 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212436 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 11:00:42.217952 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212440 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 11:00:42.217952 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212445 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 11:00:42.217952 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212449 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 11:00:42.217952 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212453 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 11:00:42.217952 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212459 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 11:00:42.217952 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212471 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 11:00:42.217952 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212476 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 11:00:42.217952 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212480 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 11:00:42.218581 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212487 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 11:00:42.218581 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212493 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 11:00:42.218581 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212498 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 11:00:42.218581 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212503 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 11:00:42.218581 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212507 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 11:00:42.218581 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212511 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 11:00:42.218581 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212515 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 11:00:42.218581 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212520 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 11:00:42.218581 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212524 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 11:00:42.218581 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212528 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 11:00:42.218581 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212532 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 11:00:42.218581 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212537 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 11:00:42.218581 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212541 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 11:00:42.218581 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212545 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 11:00:42.218581 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212549 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 11:00:42.218581 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212553 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 11:00:42.218581 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212557 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 11:00:42.218581 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212561 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 11:00:42.218581 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212565 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 11:00:42.219070 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212569 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 11:00:42.219070 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212572 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 11:00:42.219070 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212576 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 11:00:42.219070 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212579 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 11:00:42.219070 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212583 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 11:00:42.219070 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212588 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 11:00:42.219070 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212592 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 11:00:42.219070 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212596 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 11:00:42.219070 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212600 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 11:00:42.219070 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212605 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 11:00:42.219070 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212609 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 11:00:42.219070 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212621 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 11:00:42.219070 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212625 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 11:00:42.219070 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212632 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 11:00:42.219070 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212638 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 11:00:42.219513 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.212646 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 11:00:42.219513 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212909 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 11:00:42.219513 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212919 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 11:00:42.219513 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212924 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 11:00:42.219513 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212931 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 11:00:42.219513 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212938 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 11:00:42.219513 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212943 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 11:00:42.219513 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212947 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 11:00:42.219513 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212952 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 11:00:42.219513 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212956 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 11:00:42.219513 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212961 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 11:00:42.219513 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212965 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 11:00:42.219513 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212969 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 11:00:42.219513 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212974 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 11:00:42.219513 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212978 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 11:00:42.219891 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212982 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 11:00:42.219891 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212986 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 11:00:42.219891 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212990 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 11:00:42.219891 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212994 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 11:00:42.219891 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.212999 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 11:00:42.219891 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213003 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 11:00:42.219891 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213007 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 11:00:42.219891 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213011 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 11:00:42.219891 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213015 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 11:00:42.219891 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213019 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 11:00:42.219891 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213023 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 11:00:42.219891 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213028 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 11:00:42.219891 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213032 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 11:00:42.219891 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213036 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 11:00:42.219891 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213050 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 11:00:42.219891 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213054 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 11:00:42.219891 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213059 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 11:00:42.219891 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213063 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 11:00:42.219891 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213068 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 11:00:42.219891 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213072 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 11:00:42.220478 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213076 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 11:00:42.220478 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213080 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 11:00:42.220478 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213099 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 11:00:42.220478 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213104 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 11:00:42.220478 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213108 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 11:00:42.220478 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213111 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 11:00:42.220478 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213116 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 11:00:42.220478 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213119 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 11:00:42.220478 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213123 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 11:00:42.220478 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213127 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 11:00:42.220478 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213131 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 11:00:42.220478 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213135 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 11:00:42.220478 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213142 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 11:00:42.220478 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213147 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 11:00:42.220478 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213152 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 11:00:42.220478 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213157 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 11:00:42.220478 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213161 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 11:00:42.220478 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213165 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 11:00:42.220478 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213170 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 11:00:42.220987 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213174 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 11:00:42.220987 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213178 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 11:00:42.220987 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213182 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 11:00:42.220987 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213186 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 11:00:42.220987 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213190 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 11:00:42.220987 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213194 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 11:00:42.220987 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213198 2556 feature_gate.go:328] unrecognized feature gate: Example Sep 14 11:00:42.220987 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213202 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 11:00:42.220987 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213215 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 11:00:42.220987 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213219 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 11:00:42.220987 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213224 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 11:00:42.220987 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213228 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 11:00:42.220987 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213232 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 11:00:42.220987 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213237 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 11:00:42.220987 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213241 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 11:00:42.220987 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213245 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 11:00:42.220987 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213248 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 11:00:42.220987 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213253 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 11:00:42.220987 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213259 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 11:00:42.220987 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213264 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 11:00:42.221517 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213269 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 11:00:42.221517 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213274 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 11:00:42.221517 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213278 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 11:00:42.221517 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213283 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 11:00:42.221517 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213287 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 11:00:42.221517 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213292 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 11:00:42.221517 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213296 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 11:00:42.221517 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213300 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 11:00:42.221517 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213304 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 11:00:42.221517 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213309 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 11:00:42.221517 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213314 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 11:00:42.221517 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213318 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 11:00:42.221517 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213322 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 11:00:42.221517 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213327 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 11:00:42.221517 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213331 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 11:00:42.221517 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213335 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 11:00:42.221517 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213339 2556 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 11:00:42.221517 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213343 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 11:00:42.221517 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213347 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 11:00:42.221517 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213352 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 11:00:42.222053 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213356 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 11:00:42.222053 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213368 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 11:00:42.222053 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213372 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 11:00:42.222053 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213376 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 11:00:42.222053 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213380 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 11:00:42.222053 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:42.213385 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 11:00:42.222053 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.213392 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 11:00:42.222053 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.213594 2556 server.go:962] "Client rotation is on, will bootstrap in background" Sep 14 11:00:42.222053 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.217970 2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 14 11:00:42.222053 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.219119 2556 server.go:1019] "Starting client certificate rotation" Sep 14 11:00:42.222053 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.219222 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 14 11:00:42.222053 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.219293 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 14 11:00:42.222920 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.222902 2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 14 11:00:42.227364 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.227337 2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 14 11:00:42.235738 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.235720 2556 log.go:25] "Validated CRI v1 runtime API" Sep 14 11:00:42.241045 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.241027 2556 log.go:25] "Validated CRI v1 image API" Sep 14 11:00:42.241998 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.241984 2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 14 11:00:42.247333 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.247312 2556 fs.go:135] Filesystem UUIDs: map[6d00f60c-cdcf-4d5c-87d3-dd58f000c256:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c11b320b-5f2a-4c66-833a-9ff43b3ffc8d:/dev/nvme0n1p3] Sep 14 11:00:42.247392 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.247331 2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 14 11:00:42.252256 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.252151 2556 manager.go:216] Machine: {Timestamp:2026-09-14 11:00:42.250968842 +0000 UTC m=+0.314371743 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100849 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d84e4aa4af3b9f926af4fb52e4e3f SystemUUID:ec2d84e4-aa4a-f3b9-f926-af4fb52e4e3f BootID:397c1124-0f6a-4ef5-9d90-d3874edd377b Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c2:3c:23:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c2:3c:23:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:a5:81:40:7a:dd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 14 11:00:42.252256 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.252250 2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 14 11:00:42.252382 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.252366 2556 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 14 11:00:42.255219 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.255192 2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 14 11:00:42.255375 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.255221 2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-52.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 14 11:00:42.256285 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.256267 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 14 11:00:42.256637 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.256626 2556 topology_manager.go:138] "Creating topology manager with none policy" Sep 14 11:00:42.256667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.256641 2556 container_manager_linux.go:309] "Creating device plugin manager" Sep 14 11:00:42.256667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.256649 2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 14 11:00:42.257352 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.257341 2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 14 11:00:42.258126 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.258117 2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 14 11:00:42.260989 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.260975 2556 state_mem.go:36] "Initialized new in-memory state store" Sep 14 11:00:42.261149 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.261137 2556 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 14 11:00:42.261219 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.261176 2556 kubelet.go:486] "Attempting to sync node with API server" Sep 14 11:00:42.261219 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.261191 2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 14 11:00:42.261219 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.261206 2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 14 11:00:42.261341 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.261221 2556 kubelet.go:398] "Adding apiserver pod source" Sep 14 11:00:42.261341 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.261244 2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 14 11:00:42.261666 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.261651 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 14 11:00:42.261732 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.261675 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 14 11:00:42.265324 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.265252 2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 14 11:00:42.267785 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.267771 2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 14 11:00:42.267855 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.267801 2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 14 11:00:42.269288 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.269277 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 14 11:00:42.269340 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.269293 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 14 11:00:42.269340 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.269299 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 14 11:00:42.269340 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.269305 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 14 11:00:42.269340 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.269314 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 14 11:00:42.269340 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.269319 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 14 11:00:42.269340 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.269334 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 14 11:00:42.269340 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.269339 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 14 11:00:42.269340 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.269346 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 14 11:00:42.269340 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.269352 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 14 11:00:42.269610 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.269362 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 14 11:00:42.269610 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.269371 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 14 11:00:42.270067 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.270054 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 14 11:00:42.270127 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.270070 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 14 11:00:42.273329 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:42.273308 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-52.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 14 11:00:42.273405 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:42.273362 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 14 11:00:42.273854 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.273841 2556 server.go:1268] "Started kubelet" Sep 14 11:00:42.273905 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.273868 2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-52.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 14 11:00:42.273968 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.273919 2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 14 11:00:42.274707 ip-10-0-143-52 systemd[1]: Started Kubernetes Kubelet. Sep 14 11:00:42.275495 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.275479 2556 server.go:310] "Adding debug handlers to kubelet server" Sep 14 11:00:42.275976 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.275691 2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 14 11:00:42.276061 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.276000 2556 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 14 11:00:42.277161 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.277146 2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 14 11:00:42.280825 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:42.280799 2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 14 11:00:42.282238 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.282118 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 14 11:00:42.282681 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.282662 2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 14 11:00:42.282989 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:42.282967 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-52.ec2.internal\" not found" Sep 14 11:00:42.283512 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.283493 2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 14 11:00:42.283512 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.283496 2556 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 14 11:00:42.283631 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.283522 2556 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 14 11:00:42.283631 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.283530 2556 factory.go:55] Registering systemd factory Sep 14 11:00:42.283631 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.283543 2556 factory.go:223] Registration of the systemd container factory successfully Sep 14 11:00:42.283631 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.283606 2556 reconstruct.go:97] "Volume reconstruction finished" Sep 14 11:00:42.283631 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.283617 2556 reconciler.go:29] "Reconciler: start to sync state" Sep 14 11:00:42.283911 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.283899 2556 factory.go:153] Registering CRI-O factory Sep 14 11:00:42.283911 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.283912 2556 factory.go:223] Registration of the crio container factory successfully Sep 14 11:00:42.283994 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.283954 2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 14 11:00:42.283994 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.283972 2556 factory.go:103] Registering Raw factory Sep 14 11:00:42.283994 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.283988 2556 manager.go:1196] Started watching for new ooms in manager Sep 14 11:00:42.284469 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.284455 2556 manager.go:349] Starting recovery of all containers Sep 14 11:00:42.293268 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:42.293238 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 14 11:00:42.294417 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:42.294390 2556 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-52.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 14 11:00:42.294639 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:42.293318 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-52.ec2.internal.18d52af12ca014c0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-52.ec2.internal,UID:ip-10-0-143-52.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-52.ec2.internal,},FirstTimestamp:2026-09-14 11:00:42.273813696 +0000 UTC m=+0.337216600,LastTimestamp:2026-09-14 11:00:42.273813696 +0000 UTC m=+0.337216600,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-52.ec2.internal,}" Sep 14 11:00:42.298048 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.298030 2556 manager.go:354] Recovery completed Sep 14 11:00:42.302925 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.302911 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 11:00:42.304995 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.304975 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientMemory" Sep 14 11:00:42.305057 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.305008 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 11:00:42.305057 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.305023 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientPID" Sep 14 11:00:42.305603 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.305590 2556 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 14 11:00:42.305668 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.305605 2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 14 11:00:42.305668 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.305622 2556 state_mem.go:36] "Initialized new in-memory state store" Sep 14 11:00:42.305668 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.305590 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 11:00:42.306528 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.306511 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientMemory" Sep 14 11:00:42.306593 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.306543 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 11:00:42.306593 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.306557 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientPID" Sep 14 11:00:42.306864 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.306844 2556 policy_none.go:49] "None policy: Start" Sep 14 11:00:42.306903 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.306880 2556 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 14 11:00:42.306903 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.306895 2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 14 11:00:42.307130 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:42.307048 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-52.ec2.internal.18d52af12e7bd8a4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-52.ec2.internal,UID:ip-10-0-143-52.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-52.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-52.ec2.internal,},FirstTimestamp:2026-09-14 11:00:42.304993444 +0000 UTC m=+0.368396346,LastTimestamp:2026-09-14 11:00:42.304993444 +0000 UTC m=+0.368396346,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-52.ec2.internal,}" Sep 14 11:00:42.308003 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.307989 2556 policy_none.go:47] "Start" Sep 14 11:00:42.316971 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:42.316889 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-52.ec2.internal.18d52af12e7c329b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-52.ec2.internal,UID:ip-10-0-143-52.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-52.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-52.ec2.internal,},FirstTimestamp:2026-09-14 11:00:42.305016475 +0000 UTC m=+0.368419379,LastTimestamp:2026-09-14 11:00:42.305016475 +0000 UTC m=+0.368419379,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-52.ec2.internal,}" Sep 14 11:00:42.321871 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.321854 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-klk5f" Sep 14 11:00:42.349518 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:42.328962 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-52.ec2.internal.18d52af12e7c5ccf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-52.ec2.internal,UID:ip-10-0-143-52.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-52.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-52.ec2.internal,},FirstTimestamp:2026-09-14 11:00:42.305027279 +0000 UTC m=+0.368430183,LastTimestamp:2026-09-14 11:00:42.305027279 +0000 UTC m=+0.368430183,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-52.ec2.internal,}" Sep 14 11:00:42.349518 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.330856 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-klk5f" Sep 14 11:00:42.349518 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.342657 2556 manager.go:337] "Starting Device Plugin manager" Sep 14 11:00:42.349518 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:42.342702 2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 14 11:00:42.349518 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.342715 2556 server.go:87] "Starting device plugin registration server" Sep 14 11:00:42.349518 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.342962 2556 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 14 11:00:42.349518 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.342974 2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 14 11:00:42.349518 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.343111 2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 14 11:00:42.349518 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.343186 2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 14 11:00:42.349518 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.343193 2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 14 11:00:42.349518 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:42.343674 2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 14 11:00:42.349518 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:42.343728 2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-52.ec2.internal\" not found" Sep 14 11:00:42.374640 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.374610 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 14 11:00:42.375829 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.375811 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 14 11:00:42.375892 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.375870 2556 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 14 11:00:42.375943 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.375896 2556 kubelet.go:2448] "Starting kubelet main sync loop" Sep 14 11:00:42.375943 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:42.375940 2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 14 11:00:42.377861 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.377844 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 11:00:42.444054 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.444011 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 11:00:42.445069 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.445051 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientMemory" Sep 14 11:00:42.445187 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.445101 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 11:00:42.445187 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.445119 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientPID" Sep 14 11:00:42.445187 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.445149 2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-52.ec2.internal" Sep 14 11:00:42.454152 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.454081 2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-52.ec2.internal" Sep 14 11:00:42.454152 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:42.454122 2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-52.ec2.internal\": node \"ip-10-0-143-52.ec2.internal\" not found" Sep 14 11:00:42.476595 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.476567 2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-52.ec2.internal"] Sep 14 11:00:42.476707 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.476647 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 11:00:42.477553 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.477537 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientMemory" Sep 14 11:00:42.477639 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.477568 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 11:00:42.477639 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.477582 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientPID" Sep 14 11:00:42.480026 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.480012 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 11:00:42.480150 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.480135 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal" Sep 14 11:00:42.480193 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.480175 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 11:00:42.480801 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.480783 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientMemory" Sep 14 11:00:42.480801 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.480796 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientMemory" Sep 14 11:00:42.480924 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.480812 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 11:00:42.480924 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.480818 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 11:00:42.480924 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.480823 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientPID" Sep 14 11:00:42.480924 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.480831 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientPID" Sep 14 11:00:42.483050 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.483036 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-52.ec2.internal" Sep 14 11:00:42.483121 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.483064 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 11:00:42.483780 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.483765 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientMemory" Sep 14 11:00:42.483850 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.483787 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 11:00:42.483850 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.483798 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeHasSufficientPID" Sep 14 11:00:42.483949 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.483927 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c127f793063e49cd83ca3a678e448bb8-config\") pod \"kube-apiserver-proxy-ip-10-0-143-52.ec2.internal\" (UID: \"c127f793063e49cd83ca3a678e448bb8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-52.ec2.internal" Sep 14 11:00:42.484002 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.483961 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/38ba6e4ef478d7cc956f3c99e2c2415d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal\" (UID: \"38ba6e4ef478d7cc956f3c99e2c2415d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal" Sep 14 11:00:42.484002 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.483989 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38ba6e4ef478d7cc956f3c99e2c2415d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal\" (UID: \"38ba6e4ef478d7cc956f3c99e2c2415d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal" Sep 14 11:00:42.488170 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:42.488154 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-52.ec2.internal\" not found" Sep 14 11:00:42.509852 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:42.509833 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-52.ec2.internal\" not found" node="ip-10-0-143-52.ec2.internal" Sep 14 11:00:42.514402 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:42.514386 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-52.ec2.internal\" not found" node="ip-10-0-143-52.ec2.internal" Sep 14 11:00:42.584521 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.584486 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/38ba6e4ef478d7cc956f3c99e2c2415d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal\" (UID: \"38ba6e4ef478d7cc956f3c99e2c2415d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal" Sep 14 11:00:42.584521 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.584522 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38ba6e4ef478d7cc956f3c99e2c2415d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal\" (UID: \"38ba6e4ef478d7cc956f3c99e2c2415d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal" Sep 14 11:00:42.584747 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.584542 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38ba6e4ef478d7cc956f3c99e2c2415d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal\" (UID: \"38ba6e4ef478d7cc956f3c99e2c2415d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal" Sep 14 11:00:42.584747 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.584524 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/38ba6e4ef478d7cc956f3c99e2c2415d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal\" (UID: \"38ba6e4ef478d7cc956f3c99e2c2415d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal" Sep 14 11:00:42.584747 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.584548 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c127f793063e49cd83ca3a678e448bb8-config\") pod \"kube-apiserver-proxy-ip-10-0-143-52.ec2.internal\" (UID: \"c127f793063e49cd83ca3a678e448bb8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-52.ec2.internal" Sep 14 11:00:42.584747 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.584629 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c127f793063e49cd83ca3a678e448bb8-config\") pod \"kube-apiserver-proxy-ip-10-0-143-52.ec2.internal\" (UID: \"c127f793063e49cd83ca3a678e448bb8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-52.ec2.internal" Sep 14 11:00:42.588604 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:42.588586 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-52.ec2.internal\" not found" Sep 14 11:00:42.689176 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:42.689140 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-52.ec2.internal\" not found" Sep 14 11:00:42.790051 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:42.789971 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-52.ec2.internal\" not found" Sep 14 11:00:42.813155 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.813130 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal" Sep 14 11:00:42.816776 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:42.816756 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-52.ec2.internal" Sep 14 11:00:42.890877 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:42.890847 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-52.ec2.internal\" not found" Sep 14 11:00:42.991346 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:42.991319 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-52.ec2.internal\" not found" Sep 14 11:00:43.092573 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:43.092496 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-52.ec2.internal\" not found" Sep 14 11:00:43.192616 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:43.192581 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-52.ec2.internal\" not found" Sep 14 11:00:43.218811 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:43.218775 2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 14 11:00:43.219279 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:43.218930 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 14 11:00:43.293458 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:43.293284 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-52.ec2.internal\" not found" Sep 14 11:00:43.336319 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:43.336268 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-13 10:55:43 +0000 UTC" deadline="2028-05-14 07:19:40.695720056 +0000 UTC" Sep 14 11:00:43.336319 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:43.336315 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14588h18m57.359408917s" Sep 14 11:00:43.393945 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:43.393882 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-52.ec2.internal\" not found" Sep 14 11:00:43.459585 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:43.459560 2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 11:00:43.470076 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:43.470049 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc127f793063e49cd83ca3a678e448bb8.slice/crio-445c4a821e21fe3d8f345ba969cd3879654f0c98f546055c2a0da2f019ec9992 WatchSource:0}: Error finding container 445c4a821e21fe3d8f345ba969cd3879654f0c98f546055c2a0da2f019ec9992: Status 404 returned error can't find the container with id 445c4a821e21fe3d8f345ba969cd3879654f0c98f546055c2a0da2f019ec9992 Sep 14 11:00:43.470296 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:43.470273 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ba6e4ef478d7cc956f3c99e2c2415d.slice/crio-f1c393bd0462bb1dc672be6853732792ad45fb8e7bfbdf43e75afba7455d8d90 WatchSource:0}: Error finding container f1c393bd0462bb1dc672be6853732792ad45fb8e7bfbdf43e75afba7455d8d90: Status 404 returned error can't find the container with id f1c393bd0462bb1dc672be6853732792ad45fb8e7bfbdf43e75afba7455d8d90 Sep 14 11:00:43.477010 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:43.476993 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 11:00:43.483193 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:43.483173 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal" Sep 14 11:00:43.501467 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:43.501448 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 14 11:00:43.503432 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:43.503418 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-52.ec2.internal" Sep 14 11:00:43.514586 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:43.514567 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 14 11:00:43.544531 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:43.544509 2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 11:00:43.776261 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:43.776177 2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 11:00:44.264891 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.264805 2556 apiserver.go:52] "Watching apiserver" Sep 14 11:00:44.273080 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.273058 2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 14 11:00:44.273452 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.273429 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-5qhj6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal","openshift-multus/multus-k7qpb","openshift-network-diagnostics/network-check-target-d5x29","openshift-network-operator/iptables-alerter-7cn6h","openshift-ovn-kubernetes/ovnkube-node-mlmvr","kube-system/global-pull-secret-syncer-96bxc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96","openshift-cluster-node-tuning-operator/tuned-dk98m","openshift-dns/node-resolver-mcxhz","openshift-multus/multus-additional-cni-plugins-2md5p","openshift-multus/network-metrics-daemon-z75dj","kube-system/konnectivity-agent-vtv8p","kube-system/kube-apiserver-proxy-ip-10-0-143-52.ec2.internal"] Sep 14 11:00:44.277387 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.277372 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:00:44.277453 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:44.277434 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96bxc" podUID="6547f534-40c9-42c7-b118-37389bd1a650" Sep 14 11:00:44.279593 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.279574 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vtv8p" Sep 14 11:00:44.281803 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.281782 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.282199 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.282178 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 14 11:00:44.282309 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.282241 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 14 11:00:44.282381 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.282327 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 14 11:00:44.282613 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.282597 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-svjvk\"" Sep 14 11:00:44.284189 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.284158 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:00:44.284302 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:44.284232 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5x29" podUID="6de42dc8-677e-4919-816e-19952a0ca273" Sep 14 11:00:44.284735 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.284718 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 14 11:00:44.284832 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.284754 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 14 11:00:44.285935 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.285912 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 14 11:00:44.286243 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.286223 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-g9rkp\"" Sep 14 11:00:44.286334 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.286246 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 14 11:00:44.286567 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.286546 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7cn6h" Sep 14 11:00:44.289004 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.288987 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.290750 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.290733 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 14 11:00:44.290837 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.290807 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 14 11:00:44.291492 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.291476 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 14 11:00:44.291714 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.291699 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 14 11:00:44.291792 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.291704 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.292229 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.292210 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 14 11:00:44.292313 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.292245 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 14 11:00:44.292313 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.292235 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gcxrq\"" Sep 14 11:00:44.292736 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.292721 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9g28p\"" Sep 14 11:00:44.292805 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.292772 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 14 11:00:44.293077 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.293060 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 14 11:00:44.293237 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.293217 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 14 11:00:44.293775 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.293592 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnvm5\" (UniqueName: \"kubernetes.io/projected/6de42dc8-677e-4919-816e-19952a0ca273-kube-api-access-rnvm5\") pod \"network-check-target-d5x29\" (UID: \"6de42dc8-677e-4919-816e-19952a0ca273\") " pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:00:44.293775 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.293648 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6547f534-40c9-42c7-b118-37389bd1a650-original-pull-secret\") pod \"global-pull-secret-syncer-96bxc\" (UID: \"6547f534-40c9-42c7-b118-37389bd1a650\") " pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:00:44.293775 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.293686 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/624da2af-49e4-4059-9c72-20d2688683f0-cni-binary-copy\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.293775 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.293716 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-host-run-k8s-cni-cncf-io\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.294162 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.293799 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-hostroot\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.294162 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.293841 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e19a55ba-0b57-429c-a111-7b199d668301-agent-certs\") pod \"konnectivity-agent-vtv8p\" (UID: \"e19a55ba-0b57-429c-a111-7b199d668301\") " pod="kube-system/konnectivity-agent-vtv8p" Sep 14 11:00:44.294162 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.293878 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e19a55ba-0b57-429c-a111-7b199d668301-konnectivity-ca\") pod \"konnectivity-agent-vtv8p\" (UID: \"e19a55ba-0b57-429c-a111-7b199d668301\") " pod="kube-system/konnectivity-agent-vtv8p" Sep 14 11:00:44.294162 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.293907 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-multus-cni-dir\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.294162 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.293938 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-host-run-netns\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.294162 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.294009 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-host-var-lib-cni-multus\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.294162 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.294049 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-host-var-lib-kubelet\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.294508 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.294095 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5qhj6" Sep 14 11:00:44.294640 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.294197 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-multus-conf-dir\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.294692 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.294673 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6547f534-40c9-42c7-b118-37389bd1a650-dbus\") pod \"global-pull-secret-syncer-96bxc\" (UID: \"6547f534-40c9-42c7-b118-37389bd1a650\") " pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:00:44.294740 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.294709 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-system-cni-dir\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.294789 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.294738 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-cnibin\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.294789 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.294763 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/624da2af-49e4-4059-9c72-20d2688683f0-multus-daemon-config\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.294886 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.294796 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-etc-kubernetes\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.294886 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.294845 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p646g\" (UniqueName: \"kubernetes.io/projected/624da2af-49e4-4059-9c72-20d2688683f0-kube-api-access-p646g\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.294886 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.294876 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6547f534-40c9-42c7-b118-37389bd1a650-kubelet-config\") pod \"global-pull-secret-syncer-96bxc\" (UID: \"6547f534-40c9-42c7-b118-37389bd1a650\") " pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:00:44.295025 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.294907 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-host-var-lib-cni-bin\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.295025 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.294935 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-host-run-multus-certs\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.295025 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.294966 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-multus-socket-dir-parent\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.297362 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.297328 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" Sep 14 11:00:44.297473 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.297366 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 14 11:00:44.297542 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.297495 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 14 11:00:44.297784 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.297761 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dl6mn\"" Sep 14 11:00:44.297859 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.297768 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 14 11:00:44.297905 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.297858 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5c74t\"" Sep 14 11:00:44.297981 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.297965 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 14 11:00:44.298017 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.297991 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 14 11:00:44.299922 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.299904 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mcxhz" Sep 14 11:00:44.302344 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.302326 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2md5p" Sep 14 11:00:44.303147 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.303105 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 14 11:00:44.303347 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.303332 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 14 11:00:44.303719 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.303700 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 14 11:00:44.303874 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.303859 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 14 11:00:44.304170 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.304153 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zmppr\"" Sep 14 11:00:44.304396 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.304372 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 14 11:00:44.304463 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.304409 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 14 11:00:44.304726 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.304708 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:00:44.304789 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:44.304769 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z75dj" podUID="a69c5498-e86f-4c58-940f-1d14fa24a91a" Sep 14 11:00:44.305938 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.305919 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8hb8d\"" Sep 14 11:00:44.305938 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.305921 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 14 11:00:44.306660 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.306644 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 14 11:00:44.315933 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.315917 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t76lc\"" Sep 14 11:00:44.324306 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.324288 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-k8kwt" Sep 14 11:00:44.340086 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.340069 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-k8kwt" Sep 14 11:00:44.380759 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.380711 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-52.ec2.internal" event={"ID":"c127f793063e49cd83ca3a678e448bb8","Type":"ContainerStarted","Data":"445c4a821e21fe3d8f345ba969cd3879654f0c98f546055c2a0da2f019ec9992"} Sep 14 11:00:44.381953 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.381928 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal" event={"ID":"38ba6e4ef478d7cc956f3c99e2c2415d","Type":"ContainerStarted","Data":"f1c393bd0462bb1dc672be6853732792ad45fb8e7bfbdf43e75afba7455d8d90"} Sep 14 11:00:44.384748 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.384728 2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 14 11:00:44.396209 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.396186 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-run-systemd\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.396310 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.396216 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4e67edae-3518-4742-b2cd-c3d3ed2762a2-registration-dir\") pod \"aws-ebs-csi-driver-node-x9l96\" (UID: \"4e67edae-3518-4742-b2cd-c3d3ed2762a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" Sep 14 11:00:44.396310 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.396238 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6133abcd-35d6-4ac5-8a38-e88d153bdaac-system-cni-dir\") pod \"multus-additional-cni-plugins-2md5p\" (UID: \"6133abcd-35d6-4ac5-8a38-e88d153bdaac\") " pod="openshift-multus/multus-additional-cni-plugins-2md5p" Sep 14 11:00:44.396310 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.396286 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnvm5\" (UniqueName: \"kubernetes.io/projected/6de42dc8-677e-4919-816e-19952a0ca273-kube-api-access-rnvm5\") pod \"network-check-target-d5x29\" (UID: \"6de42dc8-677e-4919-816e-19952a0ca273\") " pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:00:44.396310 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.396309 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e19a55ba-0b57-429c-a111-7b199d668301-agent-certs\") pod \"konnectivity-agent-vtv8p\" (UID: \"e19a55ba-0b57-429c-a111-7b199d668301\") " pod="kube-system/konnectivity-agent-vtv8p" Sep 14 11:00:44.396465 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.396327 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-host-var-lib-cni-multus\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.396465 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.396352 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-etc-kubernetes\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.396465 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.396417 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-etc-kubernetes\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.396465 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.396420 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-host-var-lib-cni-multus\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.396670 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.396483 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-systemd-units\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.396670 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.396521 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-etc-sysctl-conf\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.396670 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.396562 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6133abcd-35d6-4ac5-8a38-e88d153bdaac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2md5p\" (UID: \"6133abcd-35d6-4ac5-8a38-e88d153bdaac\") " pod="openshift-multus/multus-additional-cni-plugins-2md5p" Sep 14 11:00:44.396670 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.396595 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-system-cni-dir\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.396670 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.396626 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p646g\" (UniqueName: \"kubernetes.io/projected/624da2af-49e4-4059-9c72-20d2688683f0-kube-api-access-p646g\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.396670 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.396653 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4259f994-0c08-408e-8db2-b3dc44beb77d-iptables-alerter-script\") pod \"iptables-alerter-7cn6h\" (UID: \"4259f994-0c08-408e-8db2-b3dc44beb77d\") " pod="openshift-network-operator/iptables-alerter-7cn6h" Sep 14 11:00:44.396880 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.396678 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpmnd\" (UniqueName: \"kubernetes.io/projected/4259f994-0c08-408e-8db2-b3dc44beb77d-kube-api-access-qpmnd\") pod \"iptables-alerter-7cn6h\" (UID: \"4259f994-0c08-408e-8db2-b3dc44beb77d\") " pod="openshift-network-operator/iptables-alerter-7cn6h" Sep 14 11:00:44.396880 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.396701 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-system-cni-dir\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.396880 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.396705 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-host-slash\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.396880 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.396753 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-etc-kubernetes\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.396880 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.396793 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h675f\" (UniqueName: \"kubernetes.io/projected/6133abcd-35d6-4ac5-8a38-e88d153bdaac-kube-api-access-h675f\") pod \"multus-additional-cni-plugins-2md5p\" (UID: \"6133abcd-35d6-4ac5-8a38-e88d153bdaac\") " pod="openshift-multus/multus-additional-cni-plugins-2md5p" Sep 14 11:00:44.396880 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.396817 2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 14 11:00:44.396880 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.396848 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-run-ovn\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.397118 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.396881 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-ovnkube-config\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.397321 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.397281 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-env-overrides\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.397418 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.397375 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jjx4\" (UniqueName: \"kubernetes.io/projected/a69c5498-e86f-4c58-940f-1d14fa24a91a-kube-api-access-2jjx4\") pod \"network-metrics-daemon-z75dj\" (UID: \"a69c5498-e86f-4c58-940f-1d14fa24a91a\") " pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:00:44.397498 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.397467 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-host-run-multus-certs\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.397581 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.397515 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-multus-conf-dir\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.397637 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.397578 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-host-run-multus-certs\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.397637 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.397579 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-host-cni-netd\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.397637 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.397597 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-multus-conf-dir\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.397803 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.397676 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb75ff85-d0ae-49a6-9f12-70361ba65c10-etc-tuned\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.397860 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.397838 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2f816499-fb81-4f59-bb3f-986f34d9c571-hosts-file\") pod \"node-resolver-mcxhz\" (UID: \"2f816499-fb81-4f59-bb3f-986f34d9c571\") " pod="openshift-dns/node-resolver-mcxhz" Sep 14 11:00:44.397914 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.397872 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-host-run-netns\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.397966 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.397911 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-host-kubelet\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.397966 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.397942 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtnpf\" (UniqueName: \"kubernetes.io/projected/eb75ff85-d0ae-49a6-9f12-70361ba65c10-kube-api-access-wtnpf\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.397966 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.397942 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-host-run-netns\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.398603 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.398572 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/27fdd2fa-1378-4960-bd86-fd1dd84516c3-serviceca\") pod \"node-ca-5qhj6\" (UID: \"27fdd2fa-1378-4960-bd86-fd1dd84516c3\") " pod="openshift-image-registry/node-ca-5qhj6" Sep 14 11:00:44.398715 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.398641 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sctbn\" (UniqueName: \"kubernetes.io/projected/27fdd2fa-1378-4960-bd86-fd1dd84516c3-kube-api-access-sctbn\") pod \"node-ca-5qhj6\" (UID: \"27fdd2fa-1378-4960-bd86-fd1dd84516c3\") " pod="openshift-image-registry/node-ca-5qhj6" Sep 14 11:00:44.398715 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.398676 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4e67edae-3518-4742-b2cd-c3d3ed2762a2-etc-selinux\") pod \"aws-ebs-csi-driver-node-x9l96\" (UID: \"4e67edae-3518-4742-b2cd-c3d3ed2762a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" Sep 14 11:00:44.398818 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.398721 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs\") pod \"network-metrics-daemon-z75dj\" (UID: \"a69c5498-e86f-4c58-940f-1d14fa24a91a\") " pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:00:44.398818 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.398748 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-host-run-ovn-kubernetes\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.398914 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.398778 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hskdl\" (UniqueName: \"kubernetes.io/projected/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-kube-api-access-hskdl\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.398914 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.398860 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-lib-modules\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.398914 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.398906 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb75ff85-d0ae-49a6-9f12-70361ba65c10-tmp\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.399059 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.398933 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f816499-fb81-4f59-bb3f-986f34d9c571-tmp-dir\") pod \"node-resolver-mcxhz\" (UID: \"2f816499-fb81-4f59-bb3f-986f34d9c571\") " pod="openshift-dns/node-resolver-mcxhz" Sep 14 11:00:44.399059 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.399002 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6133abcd-35d6-4ac5-8a38-e88d153bdaac-cnibin\") pod \"multus-additional-cni-plugins-2md5p\" (UID: \"6133abcd-35d6-4ac5-8a38-e88d153bdaac\") " pod="openshift-multus/multus-additional-cni-plugins-2md5p" Sep 14 11:00:44.399059 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.399046 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-etc-openvswitch\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.399226 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.399077 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-node-log\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.399226 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.399128 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-ovnkube-script-lib\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.399226 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.399172 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-var-lib-kubelet\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.399366 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.399234 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4e67edae-3518-4742-b2cd-c3d3ed2762a2-socket-dir\") pod \"aws-ebs-csi-driver-node-x9l96\" (UID: \"4e67edae-3518-4742-b2cd-c3d3ed2762a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" Sep 14 11:00:44.399366 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.399291 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4e67edae-3518-4742-b2cd-c3d3ed2762a2-sys-fs\") pod \"aws-ebs-csi-driver-node-x9l96\" (UID: \"4e67edae-3518-4742-b2cd-c3d3ed2762a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" Sep 14 11:00:44.399366 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.399335 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4259f994-0c08-408e-8db2-b3dc44beb77d-host-slash\") pod \"iptables-alerter-7cn6h\" (UID: \"4259f994-0c08-408e-8db2-b3dc44beb77d\") " pod="openshift-network-operator/iptables-alerter-7cn6h" Sep 14 11:00:44.399525 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.399390 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6547f534-40c9-42c7-b118-37389bd1a650-original-pull-secret\") pod \"global-pull-secret-syncer-96bxc\" (UID: \"6547f534-40c9-42c7-b118-37389bd1a650\") " pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:00:44.399525 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.399466 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-hostroot\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.399525 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.399492 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-etc-sysctl-d\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.399670 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.399537 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mgvs\" (UniqueName: \"kubernetes.io/projected/2f816499-fb81-4f59-bb3f-986f34d9c571-kube-api-access-4mgvs\") pod \"node-resolver-mcxhz\" (UID: \"2f816499-fb81-4f59-bb3f-986f34d9c571\") " pod="openshift-dns/node-resolver-mcxhz" Sep 14 11:00:44.399670 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.399595 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e19a55ba-0b57-429c-a111-7b199d668301-konnectivity-ca\") pod \"konnectivity-agent-vtv8p\" (UID: \"e19a55ba-0b57-429c-a111-7b199d668301\") " pod="kube-system/konnectivity-agent-vtv8p" Sep 14 11:00:44.399670 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.399640 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-host-var-lib-kubelet\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.399805 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.399665 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-run-openvswitch\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.399805 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.399720 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27fdd2fa-1378-4960-bd86-fd1dd84516c3-host\") pod \"node-ca-5qhj6\" (UID: \"27fdd2fa-1378-4960-bd86-fd1dd84516c3\") " pod="openshift-image-registry/node-ca-5qhj6" Sep 14 11:00:44.399805 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.399750 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4e67edae-3518-4742-b2cd-c3d3ed2762a2-device-dir\") pod \"aws-ebs-csi-driver-node-x9l96\" (UID: \"4e67edae-3518-4742-b2cd-c3d3ed2762a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" Sep 14 11:00:44.399805 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.399794 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6133abcd-35d6-4ac5-8a38-e88d153bdaac-cni-binary-copy\") pod \"multus-additional-cni-plugins-2md5p\" (UID: \"6133abcd-35d6-4ac5-8a38-e88d153bdaac\") " pod="openshift-multus/multus-additional-cni-plugins-2md5p" Sep 14 11:00:44.399997 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.399965 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6133abcd-35d6-4ac5-8a38-e88d153bdaac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2md5p\" (UID: \"6133abcd-35d6-4ac5-8a38-e88d153bdaac\") " pod="openshift-multus/multus-additional-cni-plugins-2md5p" Sep 14 11:00:44.400048 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.400027 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-run\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.400096 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.400060 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzdwf\" (UniqueName: \"kubernetes.io/projected/4e67edae-3518-4742-b2cd-c3d3ed2762a2-kube-api-access-vzdwf\") pod \"aws-ebs-csi-driver-node-x9l96\" (UID: \"4e67edae-3518-4742-b2cd-c3d3ed2762a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" Sep 14 11:00:44.400154 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.400136 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6133abcd-35d6-4ac5-8a38-e88d153bdaac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2md5p\" (UID: \"6133abcd-35d6-4ac5-8a38-e88d153bdaac\") " pod="openshift-multus/multus-additional-cni-plugins-2md5p" Sep 14 11:00:44.400712 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.400237 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6547f534-40c9-42c7-b118-37389bd1a650-kubelet-config\") pod \"global-pull-secret-syncer-96bxc\" (UID: \"6547f534-40c9-42c7-b118-37389bd1a650\") " pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:00:44.400712 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.400321 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-host-var-lib-cni-bin\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.400712 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.400373 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-var-lib-openvswitch\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.400712 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.400418 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-etc-modprobe-d\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.400712 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.400457 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-etc-sysconfig\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.400712 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.400533 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-etc-systemd\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.400712 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.400569 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-multus-socket-dir-parent\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.400712 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.400641 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/624da2af-49e4-4059-9c72-20d2688683f0-cni-binary-copy\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.401156 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.400720 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-hostroot\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.401156 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.400762 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-host-run-k8s-cni-cncf-io\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.401156 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.400824 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-host-var-lib-kubelet\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.401156 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.400919 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-host-cni-bin\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.401156 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.401008 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6547f534-40c9-42c7-b118-37389bd1a650-kubelet-config\") pod \"global-pull-secret-syncer-96bxc\" (UID: \"6547f534-40c9-42c7-b118-37389bd1a650\") " pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:00:44.401427 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.401215 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-host-var-lib-cni-bin\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.401427 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.401231 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-multus-socket-dir-parent\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.401427 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.401282 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-multus-cni-dir\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.401427 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.401287 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-host-run-k8s-cni-cncf-io\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.401427 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.401385 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-host-run-netns\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.401718 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.401432 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-log-socket\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.401718 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.401499 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-multus-cni-dir\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.401718 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.401557 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-ovn-node-metrics-cert\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.401718 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.401597 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-host\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.401718 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.401638 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4e67edae-3518-4742-b2cd-c3d3ed2762a2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x9l96\" (UID: \"4e67edae-3518-4742-b2cd-c3d3ed2762a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" Sep 14 11:00:44.401718 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.401687 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6547f534-40c9-42c7-b118-37389bd1a650-dbus\") pod \"global-pull-secret-syncer-96bxc\" (UID: \"6547f534-40c9-42c7-b118-37389bd1a650\") " pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:00:44.401980 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.401756 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-cnibin\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.401980 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.401761 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6547f534-40c9-42c7-b118-37389bd1a650-dbus\") pod \"global-pull-secret-syncer-96bxc\" (UID: \"6547f534-40c9-42c7-b118-37389bd1a650\") " pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:00:44.401980 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.401812 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/624da2af-49e4-4059-9c72-20d2688683f0-multus-daemon-config\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.401980 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.401821 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/624da2af-49e4-4059-9c72-20d2688683f0-cnibin\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.402199 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.401976 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.402199 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:44.401990 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 11:00:44.402199 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.402004 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/624da2af-49e4-4059-9c72-20d2688683f0-cni-binary-copy\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.402199 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.402081 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-sys\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.402199 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:44.402157 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6547f534-40c9-42c7-b118-37389bd1a650-original-pull-secret podName:6547f534-40c9-42c7-b118-37389bd1a650 nodeName:}" failed. No retries permitted until 2026-09-14 11:00:44.902132521 +0000 UTC m=+2.965535430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6547f534-40c9-42c7-b118-37389bd1a650-original-pull-secret") pod "global-pull-secret-syncer-96bxc" (UID: "6547f534-40c9-42c7-b118-37389bd1a650") : object "kube-system"/"original-pull-secret" not registered Sep 14 11:00:44.402486 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.402369 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e19a55ba-0b57-429c-a111-7b199d668301-agent-certs\") pod \"konnectivity-agent-vtv8p\" (UID: \"e19a55ba-0b57-429c-a111-7b199d668301\") " pod="kube-system/konnectivity-agent-vtv8p" Sep 14 11:00:44.402539 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.402493 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/624da2af-49e4-4059-9c72-20d2688683f0-multus-daemon-config\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.402539 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.402522 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e19a55ba-0b57-429c-a111-7b199d668301-konnectivity-ca\") pod \"konnectivity-agent-vtv8p\" (UID: \"e19a55ba-0b57-429c-a111-7b199d668301\") " pod="kube-system/konnectivity-agent-vtv8p" Sep 14 11:00:44.403069 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:44.403053 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 11:00:44.403137 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:44.403074 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 11:00:44.403137 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:44.403085 2556 projected.go:196] Error preparing data for projected volume kube-api-access-rnvm5 for pod openshift-network-diagnostics/network-check-target-d5x29: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:00:44.403211 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:44.403156 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6de42dc8-677e-4919-816e-19952a0ca273-kube-api-access-rnvm5 podName:6de42dc8-677e-4919-816e-19952a0ca273 nodeName:}" failed. No retries permitted until 2026-09-14 11:00:44.903143133 +0000 UTC m=+2.966546024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rnvm5" (UniqueName: "kubernetes.io/projected/6de42dc8-677e-4919-816e-19952a0ca273-kube-api-access-rnvm5") pod "network-check-target-d5x29" (UID: "6de42dc8-677e-4919-816e-19952a0ca273") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:00:44.405960 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.405940 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p646g\" (UniqueName: \"kubernetes.io/projected/624da2af-49e4-4059-9c72-20d2688683f0-kube-api-access-p646g\") pod \"multus-k7qpb\" (UID: \"624da2af-49e4-4059-9c72-20d2688683f0\") " pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.503246 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503211 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-host-cni-bin\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.503246 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503245 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-host-run-netns\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.503519 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503261 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-log-socket\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.503519 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503278 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-ovn-node-metrics-cert\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.503519 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503295 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-host\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.503519 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503311 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4e67edae-3518-4742-b2cd-c3d3ed2762a2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x9l96\" (UID: \"4e67edae-3518-4742-b2cd-c3d3ed2762a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" Sep 14 11:00:44.503519 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503358 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-log-socket\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.503519 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503383 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.503519 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503365 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-host-run-netns\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.503519 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503402 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-host\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.503519 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503412 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4e67edae-3518-4742-b2cd-c3d3ed2762a2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x9l96\" (UID: \"4e67edae-3518-4742-b2cd-c3d3ed2762a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" Sep 14 11:00:44.503519 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503425 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-sys\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.503519 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503439 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.503519 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503457 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-host-cni-bin\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.503519 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503453 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-run-systemd\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.503519 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503492 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-sys\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.503519 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503506 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4e67edae-3518-4742-b2cd-c3d3ed2762a2-registration-dir\") pod \"aws-ebs-csi-driver-node-x9l96\" (UID: \"4e67edae-3518-4742-b2cd-c3d3ed2762a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" Sep 14 11:00:44.504372 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503537 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-run-systemd\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.504372 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503545 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6133abcd-35d6-4ac5-8a38-e88d153bdaac-system-cni-dir\") pod \"multus-additional-cni-plugins-2md5p\" (UID: \"6133abcd-35d6-4ac5-8a38-e88d153bdaac\") " pod="openshift-multus/multus-additional-cni-plugins-2md5p" Sep 14 11:00:44.504372 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503571 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6133abcd-35d6-4ac5-8a38-e88d153bdaac-system-cni-dir\") pod \"multus-additional-cni-plugins-2md5p\" (UID: \"6133abcd-35d6-4ac5-8a38-e88d153bdaac\") " pod="openshift-multus/multus-additional-cni-plugins-2md5p" Sep 14 11:00:44.504372 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503578 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4e67edae-3518-4742-b2cd-c3d3ed2762a2-registration-dir\") pod \"aws-ebs-csi-driver-node-x9l96\" (UID: \"4e67edae-3518-4742-b2cd-c3d3ed2762a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" Sep 14 11:00:44.504372 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503592 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-systemd-units\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.504372 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503627 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-etc-sysctl-conf\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.504372 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503652 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6133abcd-35d6-4ac5-8a38-e88d153bdaac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2md5p\" (UID: \"6133abcd-35d6-4ac5-8a38-e88d153bdaac\") " pod="openshift-multus/multus-additional-cni-plugins-2md5p" Sep 14 11:00:44.504372 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503666 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-systemd-units\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.504372 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503693 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4259f994-0c08-408e-8db2-b3dc44beb77d-iptables-alerter-script\") pod \"iptables-alerter-7cn6h\" (UID: \"4259f994-0c08-408e-8db2-b3dc44beb77d\") " pod="openshift-network-operator/iptables-alerter-7cn6h" Sep 14 11:00:44.504372 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503714 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpmnd\" (UniqueName: \"kubernetes.io/projected/4259f994-0c08-408e-8db2-b3dc44beb77d-kube-api-access-qpmnd\") pod \"iptables-alerter-7cn6h\" (UID: \"4259f994-0c08-408e-8db2-b3dc44beb77d\") " pod="openshift-network-operator/iptables-alerter-7cn6h" Sep 14 11:00:44.504372 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503758 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-host-slash\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.504372 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503782 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-etc-kubernetes\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.504372 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503791 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-etc-sysctl-conf\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.504372 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503809 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h675f\" (UniqueName: \"kubernetes.io/projected/6133abcd-35d6-4ac5-8a38-e88d153bdaac-kube-api-access-h675f\") pod \"multus-additional-cni-plugins-2md5p\" (UID: \"6133abcd-35d6-4ac5-8a38-e88d153bdaac\") " pod="openshift-multus/multus-additional-cni-plugins-2md5p" Sep 14 11:00:44.504372 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503823 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6133abcd-35d6-4ac5-8a38-e88d153bdaac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2md5p\" (UID: \"6133abcd-35d6-4ac5-8a38-e88d153bdaac\") " pod="openshift-multus/multus-additional-cni-plugins-2md5p" Sep 14 11:00:44.504372 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503850 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-host-slash\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.504372 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503903 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-etc-kubernetes\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.505288 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503941 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-run-ovn\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.505288 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503969 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-ovnkube-config\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.505288 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.503984 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-run-ovn\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.505288 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504016 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-env-overrides\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.505288 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504061 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jjx4\" (UniqueName: \"kubernetes.io/projected/a69c5498-e86f-4c58-940f-1d14fa24a91a-kube-api-access-2jjx4\") pod \"network-metrics-daemon-z75dj\" (UID: \"a69c5498-e86f-4c58-940f-1d14fa24a91a\") " pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:00:44.505288 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504104 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-host-cni-netd\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.505288 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504130 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb75ff85-d0ae-49a6-9f12-70361ba65c10-etc-tuned\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.505288 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504157 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2f816499-fb81-4f59-bb3f-986f34d9c571-hosts-file\") pod \"node-resolver-mcxhz\" (UID: \"2f816499-fb81-4f59-bb3f-986f34d9c571\") " pod="openshift-dns/node-resolver-mcxhz" Sep 14 11:00:44.505288 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504211 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-host-kubelet\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.505288 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504257 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtnpf\" (UniqueName: \"kubernetes.io/projected/eb75ff85-d0ae-49a6-9f12-70361ba65c10-kube-api-access-wtnpf\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.505288 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504267 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2f816499-fb81-4f59-bb3f-986f34d9c571-hosts-file\") pod \"node-resolver-mcxhz\" (UID: \"2f816499-fb81-4f59-bb3f-986f34d9c571\") " pod="openshift-dns/node-resolver-mcxhz" Sep 14 11:00:44.505288 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504285 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/27fdd2fa-1378-4960-bd86-fd1dd84516c3-serviceca\") pod \"node-ca-5qhj6\" (UID: \"27fdd2fa-1378-4960-bd86-fd1dd84516c3\") " pod="openshift-image-registry/node-ca-5qhj6" Sep 14 11:00:44.505288 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504318 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sctbn\" (UniqueName: \"kubernetes.io/projected/27fdd2fa-1378-4960-bd86-fd1dd84516c3-kube-api-access-sctbn\") pod \"node-ca-5qhj6\" (UID: \"27fdd2fa-1378-4960-bd86-fd1dd84516c3\") " pod="openshift-image-registry/node-ca-5qhj6" Sep 14 11:00:44.505288 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504344 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4e67edae-3518-4742-b2cd-c3d3ed2762a2-etc-selinux\") pod \"aws-ebs-csi-driver-node-x9l96\" (UID: \"4e67edae-3518-4742-b2cd-c3d3ed2762a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" Sep 14 11:00:44.505288 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504389 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4259f994-0c08-408e-8db2-b3dc44beb77d-iptables-alerter-script\") pod \"iptables-alerter-7cn6h\" (UID: \"4259f994-0c08-408e-8db2-b3dc44beb77d\") " pod="openshift-network-operator/iptables-alerter-7cn6h" Sep 14 11:00:44.505288 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504411 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs\") pod \"network-metrics-daemon-z75dj\" (UID: \"a69c5498-e86f-4c58-940f-1d14fa24a91a\") " pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:00:44.505288 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504454 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-host-run-ovn-kubernetes\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.506211 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504483 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hskdl\" (UniqueName: \"kubernetes.io/projected/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-kube-api-access-hskdl\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.506211 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504512 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-lib-modules\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.506211 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504519 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-env-overrides\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.506211 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504536 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb75ff85-d0ae-49a6-9f12-70361ba65c10-tmp\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.506211 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504561 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f816499-fb81-4f59-bb3f-986f34d9c571-tmp-dir\") pod \"node-resolver-mcxhz\" (UID: \"2f816499-fb81-4f59-bb3f-986f34d9c571\") " pod="openshift-dns/node-resolver-mcxhz" Sep 14 11:00:44.506211 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504586 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6133abcd-35d6-4ac5-8a38-e88d153bdaac-cnibin\") pod \"multus-additional-cni-plugins-2md5p\" (UID: \"6133abcd-35d6-4ac5-8a38-e88d153bdaac\") " pod="openshift-multus/multus-additional-cni-plugins-2md5p" Sep 14 11:00:44.506211 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504588 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-host-run-ovn-kubernetes\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.506211 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504646 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-ovnkube-config\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.506211 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504693 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4e67edae-3518-4742-b2cd-c3d3ed2762a2-etc-selinux\") pod \"aws-ebs-csi-driver-node-x9l96\" (UID: \"4e67edae-3518-4742-b2cd-c3d3ed2762a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" Sep 14 11:00:44.506211 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504701 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-host-kubelet\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.506211 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504208 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-host-cni-netd\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.506211 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504727 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-etc-openvswitch\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.506211 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:44.504745 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:00:44.506211 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504770 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-etc-openvswitch\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.506211 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:44.504815 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs podName:a69c5498-e86f-4c58-940f-1d14fa24a91a nodeName:}" failed. No retries permitted until 2026-09-14 11:00:45.004795972 +0000 UTC m=+3.068198882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs") pod "network-metrics-daemon-z75dj" (UID: "a69c5498-e86f-4c58-940f-1d14fa24a91a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:00:44.506211 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504822 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6133abcd-35d6-4ac5-8a38-e88d153bdaac-cnibin\") pod \"multus-additional-cni-plugins-2md5p\" (UID: \"6133abcd-35d6-4ac5-8a38-e88d153bdaac\") " pod="openshift-multus/multus-additional-cni-plugins-2md5p" Sep 14 11:00:44.506211 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504807 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-lib-modules\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.506739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504861 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-node-log\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.506739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504921 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-ovnkube-script-lib\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.506739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504949 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-var-lib-kubelet\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.506739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504959 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-node-log\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.506739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.504991 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4e67edae-3518-4742-b2cd-c3d3ed2762a2-socket-dir\") pod \"aws-ebs-csi-driver-node-x9l96\" (UID: \"4e67edae-3518-4742-b2cd-c3d3ed2762a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" Sep 14 11:00:44.506739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505019 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4e67edae-3518-4742-b2cd-c3d3ed2762a2-sys-fs\") pod \"aws-ebs-csi-driver-node-x9l96\" (UID: \"4e67edae-3518-4742-b2cd-c3d3ed2762a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" Sep 14 11:00:44.506739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505045 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4259f994-0c08-408e-8db2-b3dc44beb77d-host-slash\") pod \"iptables-alerter-7cn6h\" (UID: \"4259f994-0c08-408e-8db2-b3dc44beb77d\") " pod="openshift-network-operator/iptables-alerter-7cn6h" Sep 14 11:00:44.506739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505054 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/27fdd2fa-1378-4960-bd86-fd1dd84516c3-serviceca\") pod \"node-ca-5qhj6\" (UID: \"27fdd2fa-1378-4960-bd86-fd1dd84516c3\") " pod="openshift-image-registry/node-ca-5qhj6" Sep 14 11:00:44.506739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505022 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-var-lib-kubelet\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.506739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505087 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-etc-sysctl-d\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.506739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505124 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mgvs\" (UniqueName: \"kubernetes.io/projected/2f816499-fb81-4f59-bb3f-986f34d9c571-kube-api-access-4mgvs\") pod \"node-resolver-mcxhz\" (UID: \"2f816499-fb81-4f59-bb3f-986f34d9c571\") " pod="openshift-dns/node-resolver-mcxhz" Sep 14 11:00:44.506739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505141 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4e67edae-3518-4742-b2cd-c3d3ed2762a2-socket-dir\") pod \"aws-ebs-csi-driver-node-x9l96\" (UID: \"4e67edae-3518-4742-b2cd-c3d3ed2762a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" Sep 14 11:00:44.506739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505155 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-run-openvswitch\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.506739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505198 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27fdd2fa-1378-4960-bd86-fd1dd84516c3-host\") pod \"node-ca-5qhj6\" (UID: \"27fdd2fa-1378-4960-bd86-fd1dd84516c3\") " pod="openshift-image-registry/node-ca-5qhj6" Sep 14 11:00:44.506739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505217 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-etc-sysctl-d\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.506739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505222 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4259f994-0c08-408e-8db2-b3dc44beb77d-host-slash\") pod \"iptables-alerter-7cn6h\" (UID: \"4259f994-0c08-408e-8db2-b3dc44beb77d\") " pod="openshift-network-operator/iptables-alerter-7cn6h" Sep 14 11:00:44.506739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505226 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4e67edae-3518-4742-b2cd-c3d3ed2762a2-device-dir\") pod \"aws-ebs-csi-driver-node-x9l96\" (UID: \"4e67edae-3518-4742-b2cd-c3d3ed2762a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" Sep 14 11:00:44.507400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505170 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4e67edae-3518-4742-b2cd-c3d3ed2762a2-sys-fs\") pod \"aws-ebs-csi-driver-node-x9l96\" (UID: \"4e67edae-3518-4742-b2cd-c3d3ed2762a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" Sep 14 11:00:44.507400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505263 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6133abcd-35d6-4ac5-8a38-e88d153bdaac-cni-binary-copy\") pod \"multus-additional-cni-plugins-2md5p\" (UID: \"6133abcd-35d6-4ac5-8a38-e88d153bdaac\") " pod="openshift-multus/multus-additional-cni-plugins-2md5p" Sep 14 11:00:44.507400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505277 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4e67edae-3518-4742-b2cd-c3d3ed2762a2-device-dir\") pod \"aws-ebs-csi-driver-node-x9l96\" (UID: \"4e67edae-3518-4742-b2cd-c3d3ed2762a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" Sep 14 11:00:44.507400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505297 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6133abcd-35d6-4ac5-8a38-e88d153bdaac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2md5p\" (UID: \"6133abcd-35d6-4ac5-8a38-e88d153bdaac\") " pod="openshift-multus/multus-additional-cni-plugins-2md5p" Sep 14 11:00:44.507400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505318 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27fdd2fa-1378-4960-bd86-fd1dd84516c3-host\") pod \"node-ca-5qhj6\" (UID: \"27fdd2fa-1378-4960-bd86-fd1dd84516c3\") " pod="openshift-image-registry/node-ca-5qhj6" Sep 14 11:00:44.507400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505324 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-run\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.507400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505279 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-run-openvswitch\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.507400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505375 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzdwf\" (UniqueName: \"kubernetes.io/projected/4e67edae-3518-4742-b2cd-c3d3ed2762a2-kube-api-access-vzdwf\") pod \"aws-ebs-csi-driver-node-x9l96\" (UID: \"4e67edae-3518-4742-b2cd-c3d3ed2762a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" Sep 14 11:00:44.507400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505402 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6133abcd-35d6-4ac5-8a38-e88d153bdaac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2md5p\" (UID: \"6133abcd-35d6-4ac5-8a38-e88d153bdaac\") " pod="openshift-multus/multus-additional-cni-plugins-2md5p" Sep 14 11:00:44.507400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505433 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-var-lib-openvswitch\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.507400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505452 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-run\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.507400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505459 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-etc-modprobe-d\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.507400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505489 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-etc-sysconfig\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.507400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505516 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-etc-systemd\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.507400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505519 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-ovnkube-script-lib\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.507400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505593 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-var-lib-openvswitch\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.507400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505648 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-etc-modprobe-d\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.508001 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505691 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f816499-fb81-4f59-bb3f-986f34d9c571-tmp-dir\") pod \"node-resolver-mcxhz\" (UID: \"2f816499-fb81-4f59-bb3f-986f34d9c571\") " pod="openshift-dns/node-resolver-mcxhz" Sep 14 11:00:44.508001 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505777 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-etc-sysconfig\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.508001 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505834 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb75ff85-d0ae-49a6-9f12-70361ba65c10-etc-systemd\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.508001 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.505851 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6133abcd-35d6-4ac5-8a38-e88d153bdaac-cni-binary-copy\") pod \"multus-additional-cni-plugins-2md5p\" (UID: \"6133abcd-35d6-4ac5-8a38-e88d153bdaac\") " pod="openshift-multus/multus-additional-cni-plugins-2md5p" Sep 14 11:00:44.508001 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.506323 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-ovn-node-metrics-cert\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.508001 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.506342 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6133abcd-35d6-4ac5-8a38-e88d153bdaac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2md5p\" (UID: \"6133abcd-35d6-4ac5-8a38-e88d153bdaac\") " pod="openshift-multus/multus-additional-cni-plugins-2md5p" Sep 14 11:00:44.508001 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.506364 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6133abcd-35d6-4ac5-8a38-e88d153bdaac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2md5p\" (UID: \"6133abcd-35d6-4ac5-8a38-e88d153bdaac\") " pod="openshift-multus/multus-additional-cni-plugins-2md5p" Sep 14 11:00:44.508001 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.506886 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb75ff85-d0ae-49a6-9f12-70361ba65c10-etc-tuned\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.508001 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.507641 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb75ff85-d0ae-49a6-9f12-70361ba65c10-tmp\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.516006 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.515791 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mgvs\" (UniqueName: \"kubernetes.io/projected/2f816499-fb81-4f59-bb3f-986f34d9c571-kube-api-access-4mgvs\") pod \"node-resolver-mcxhz\" (UID: \"2f816499-fb81-4f59-bb3f-986f34d9c571\") " pod="openshift-dns/node-resolver-mcxhz" Sep 14 11:00:44.516105 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.516015 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jjx4\" (UniqueName: \"kubernetes.io/projected/a69c5498-e86f-4c58-940f-1d14fa24a91a-kube-api-access-2jjx4\") pod \"network-metrics-daemon-z75dj\" (UID: \"a69c5498-e86f-4c58-940f-1d14fa24a91a\") " pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:00:44.516260 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.516236 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hskdl\" (UniqueName: \"kubernetes.io/projected/4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3-kube-api-access-hskdl\") pod \"ovnkube-node-mlmvr\" (UID: \"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.516437 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.516413 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtnpf\" (UniqueName: \"kubernetes.io/projected/eb75ff85-d0ae-49a6-9f12-70361ba65c10-kube-api-access-wtnpf\") pod \"tuned-dk98m\" (UID: \"eb75ff85-d0ae-49a6-9f12-70361ba65c10\") " pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.516576 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.516548 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpmnd\" (UniqueName: \"kubernetes.io/projected/4259f994-0c08-408e-8db2-b3dc44beb77d-kube-api-access-qpmnd\") pod \"iptables-alerter-7cn6h\" (UID: \"4259f994-0c08-408e-8db2-b3dc44beb77d\") " pod="openshift-network-operator/iptables-alerter-7cn6h" Sep 14 11:00:44.516668 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.516652 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzdwf\" (UniqueName: \"kubernetes.io/projected/4e67edae-3518-4742-b2cd-c3d3ed2762a2-kube-api-access-vzdwf\") pod \"aws-ebs-csi-driver-node-x9l96\" (UID: \"4e67edae-3518-4742-b2cd-c3d3ed2762a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" Sep 14 11:00:44.517476 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.517455 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h675f\" (UniqueName: \"kubernetes.io/projected/6133abcd-35d6-4ac5-8a38-e88d153bdaac-kube-api-access-h675f\") pod \"multus-additional-cni-plugins-2md5p\" (UID: \"6133abcd-35d6-4ac5-8a38-e88d153bdaac\") " pod="openshift-multus/multus-additional-cni-plugins-2md5p" Sep 14 11:00:44.519013 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.518995 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sctbn\" (UniqueName: \"kubernetes.io/projected/27fdd2fa-1378-4960-bd86-fd1dd84516c3-kube-api-access-sctbn\") pod \"node-ca-5qhj6\" (UID: \"27fdd2fa-1378-4960-bd86-fd1dd84516c3\") " pod="openshift-image-registry/node-ca-5qhj6" Sep 14 11:00:44.587400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.587370 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 11:00:44.590308 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.590276 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vtv8p" Sep 14 11:00:44.599496 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.599471 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k7qpb" Sep 14 11:00:44.602386 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:44.602352 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19a55ba_0b57_429c_a111_7b199d668301.slice/crio-3f380ddd19a9ec228e597aba7813fc5a2a3be271abe171ed82dc148478999c99 WatchSource:0}: Error finding container 3f380ddd19a9ec228e597aba7813fc5a2a3be271abe171ed82dc148478999c99: Status 404 returned error can't find the container with id 3f380ddd19a9ec228e597aba7813fc5a2a3be271abe171ed82dc148478999c99 Sep 14 11:00:44.606321 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.606299 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7cn6h" Sep 14 11:00:44.609860 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:44.609829 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod624da2af_49e4_4059_9c72_20d2688683f0.slice/crio-61fc6e7a00e98022fc8e0da965db8340cc3ea23a85695b100b4e2e52f0e9a503 WatchSource:0}: Error finding container 61fc6e7a00e98022fc8e0da965db8340cc3ea23a85695b100b4e2e52f0e9a503: Status 404 returned error can't find the container with id 61fc6e7a00e98022fc8e0da965db8340cc3ea23a85695b100b4e2e52f0e9a503 Sep 14 11:00:44.612600 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.612430 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:00:44.616899 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.616876 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dk98m" Sep 14 11:00:44.620823 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:44.620794 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4259f994_0c08_408e_8db2_b3dc44beb77d.slice/crio-13a1f25baa80ea95aa19ffb38b7653cac2ff9ff85bc27a568625be80cfb60013 WatchSource:0}: Error finding container 13a1f25baa80ea95aa19ffb38b7653cac2ff9ff85bc27a568625be80cfb60013: Status 404 returned error can't find the container with id 13a1f25baa80ea95aa19ffb38b7653cac2ff9ff85bc27a568625be80cfb60013 Sep 14 11:00:44.623147 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.623125 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5qhj6" Sep 14 11:00:44.626338 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:44.626306 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf809e2_5d44_4a24_a5d9_58e6e49ab7a3.slice/crio-d0ee20f9f6226efde72aecc10826a509132e900404a6a87f66e49d23bef6b0af WatchSource:0}: Error finding container d0ee20f9f6226efde72aecc10826a509132e900404a6a87f66e49d23bef6b0af: Status 404 returned error can't find the container with id d0ee20f9f6226efde72aecc10826a509132e900404a6a87f66e49d23bef6b0af Sep 14 11:00:44.631633 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.631470 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" Sep 14 11:00:44.634151 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.634130 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mcxhz" Sep 14 11:00:44.637252 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:44.637223 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27fdd2fa_1378_4960_bd86_fd1dd84516c3.slice/crio-928da4627c636963b55ef89807f91916cfd712d076c995e4de8367cec179fb68 WatchSource:0}: Error finding container 928da4627c636963b55ef89807f91916cfd712d076c995e4de8367cec179fb68: Status 404 returned error can't find the container with id 928da4627c636963b55ef89807f91916cfd712d076c995e4de8367cec179fb68 Sep 14 11:00:44.639312 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.639292 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2md5p" Sep 14 11:00:44.648062 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:44.647763 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e67edae_3518_4742_b2cd_c3d3ed2762a2.slice/crio-67d5b20f602c61cecfc4c355482dd1521bab9bb9225c6e15a29f034a755de27a WatchSource:0}: Error finding container 67d5b20f602c61cecfc4c355482dd1521bab9bb9225c6e15a29f034a755de27a: Status 404 returned error can't find the container with id 67d5b20f602c61cecfc4c355482dd1521bab9bb9225c6e15a29f034a755de27a Sep 14 11:00:44.653895 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:44.653860 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6133abcd_35d6_4ac5_8a38_e88d153bdaac.slice/crio-652678bc39dc1ea7c51990892a66600fd1dca3292261aede3711ec0b7c6a6155 WatchSource:0}: Error finding container 652678bc39dc1ea7c51990892a66600fd1dca3292261aede3711ec0b7c6a6155: Status 404 returned error can't find the container with id 652678bc39dc1ea7c51990892a66600fd1dca3292261aede3711ec0b7c6a6155 Sep 14 11:00:44.655743 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:00:44.655717 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f816499_fb81_4f59_bb3f_986f34d9c571.slice/crio-12ae4874745bf823f8fe4ac012e66e03e8629a8d4290dfc7dfa4315457e6f2f7 WatchSource:0}: Error finding container 12ae4874745bf823f8fe4ac012e66e03e8629a8d4290dfc7dfa4315457e6f2f7: Status 404 returned error can't find the container with id 12ae4874745bf823f8fe4ac012e66e03e8629a8d4290dfc7dfa4315457e6f2f7 Sep 14 11:00:44.908799 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.908695 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnvm5\" (UniqueName: \"kubernetes.io/projected/6de42dc8-677e-4919-816e-19952a0ca273-kube-api-access-rnvm5\") pod \"network-check-target-d5x29\" (UID: \"6de42dc8-677e-4919-816e-19952a0ca273\") " pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:00:44.908973 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:44.908800 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6547f534-40c9-42c7-b118-37389bd1a650-original-pull-secret\") pod \"global-pull-secret-syncer-96bxc\" (UID: \"6547f534-40c9-42c7-b118-37389bd1a650\") " pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:00:44.909049 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:44.908969 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 11:00:44.909049 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:44.908957 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 11:00:44.909049 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:44.909037 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6547f534-40c9-42c7-b118-37389bd1a650-original-pull-secret podName:6547f534-40c9-42c7-b118-37389bd1a650 nodeName:}" failed. No retries permitted until 2026-09-14 11:00:45.909019111 +0000 UTC m=+3.972422006 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6547f534-40c9-42c7-b118-37389bd1a650-original-pull-secret") pod "global-pull-secret-syncer-96bxc" (UID: "6547f534-40c9-42c7-b118-37389bd1a650") : object "kube-system"/"original-pull-secret" not registered Sep 14 11:00:44.909240 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:44.909051 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 11:00:44.909240 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:44.909067 2556 projected.go:196] Error preparing data for projected volume kube-api-access-rnvm5 for pod openshift-network-diagnostics/network-check-target-d5x29: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:00:44.909240 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:44.909151 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6de42dc8-677e-4919-816e-19952a0ca273-kube-api-access-rnvm5 podName:6de42dc8-677e-4919-816e-19952a0ca273 nodeName:}" failed. No retries permitted until 2026-09-14 11:00:45.909107452 +0000 UTC m=+3.972510358 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rnvm5" (UniqueName: "kubernetes.io/projected/6de42dc8-677e-4919-816e-19952a0ca273-kube-api-access-rnvm5") pod "network-check-target-d5x29" (UID: "6de42dc8-677e-4919-816e-19952a0ca273") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:00:45.009338 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:45.009292 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs\") pod \"network-metrics-daemon-z75dj\" (UID: \"a69c5498-e86f-4c58-940f-1d14fa24a91a\") " pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:00:45.009534 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:45.009493 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:00:45.009601 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:45.009579 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs podName:a69c5498-e86f-4c58-940f-1d14fa24a91a nodeName:}" failed. No retries permitted until 2026-09-14 11:00:46.009558797 +0000 UTC m=+4.072961699 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs") pod "network-metrics-daemon-z75dj" (UID: "a69c5498-e86f-4c58-940f-1d14fa24a91a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:00:45.343591 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:45.343451 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 10:55:45 +0000 UTC" deadline="2028-05-31 10:26:06.304638874 +0000 UTC" Sep 14 11:00:45.343591 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:45.343521 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14999h25m20.961122207s" Sep 14 11:00:45.390336 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:45.390108 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k7qpb" event={"ID":"624da2af-49e4-4059-9c72-20d2688683f0","Type":"ContainerStarted","Data":"61fc6e7a00e98022fc8e0da965db8340cc3ea23a85695b100b4e2e52f0e9a503"} Sep 14 11:00:45.392726 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:45.392694 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vtv8p" event={"ID":"e19a55ba-0b57-429c-a111-7b199d668301","Type":"ContainerStarted","Data":"3f380ddd19a9ec228e597aba7813fc5a2a3be271abe171ed82dc148478999c99"} Sep 14 11:00:45.394889 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:45.394844 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mcxhz" event={"ID":"2f816499-fb81-4f59-bb3f-986f34d9c571","Type":"ContainerStarted","Data":"12ae4874745bf823f8fe4ac012e66e03e8629a8d4290dfc7dfa4315457e6f2f7"} Sep 14 11:00:45.396822 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:45.396800 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2md5p" event={"ID":"6133abcd-35d6-4ac5-8a38-e88d153bdaac","Type":"ContainerStarted","Data":"652678bc39dc1ea7c51990892a66600fd1dca3292261aede3711ec0b7c6a6155"} Sep 14 11:00:45.398694 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:45.398665 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" event={"ID":"4e67edae-3518-4742-b2cd-c3d3ed2762a2","Type":"ContainerStarted","Data":"67d5b20f602c61cecfc4c355482dd1521bab9bb9225c6e15a29f034a755de27a"} Sep 14 11:00:45.401422 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:45.401397 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dk98m" event={"ID":"eb75ff85-d0ae-49a6-9f12-70361ba65c10","Type":"ContainerStarted","Data":"d5d97ad3820e93ac42a5999c7086509b6fb048cfe64e602124b35a374cfbb376"} Sep 14 11:00:45.409421 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:45.409393 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7cn6h" event={"ID":"4259f994-0c08-408e-8db2-b3dc44beb77d","Type":"ContainerStarted","Data":"13a1f25baa80ea95aa19ffb38b7653cac2ff9ff85bc27a568625be80cfb60013"} Sep 14 11:00:45.412184 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:45.412103 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5qhj6" event={"ID":"27fdd2fa-1378-4960-bd86-fd1dd84516c3","Type":"ContainerStarted","Data":"928da4627c636963b55ef89807f91916cfd712d076c995e4de8367cec179fb68"} Sep 14 11:00:45.415421 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:45.415296 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" event={"ID":"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3","Type":"ContainerStarted","Data":"d0ee20f9f6226efde72aecc10826a509132e900404a6a87f66e49d23bef6b0af"} Sep 14 11:00:45.919077 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:45.918243 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnvm5\" (UniqueName: \"kubernetes.io/projected/6de42dc8-677e-4919-816e-19952a0ca273-kube-api-access-rnvm5\") pod \"network-check-target-d5x29\" (UID: \"6de42dc8-677e-4919-816e-19952a0ca273\") " pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:00:45.919077 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:45.918333 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6547f534-40c9-42c7-b118-37389bd1a650-original-pull-secret\") pod \"global-pull-secret-syncer-96bxc\" (UID: \"6547f534-40c9-42c7-b118-37389bd1a650\") " pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:00:45.919077 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:45.918481 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 11:00:45.919077 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:45.918546 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6547f534-40c9-42c7-b118-37389bd1a650-original-pull-secret podName:6547f534-40c9-42c7-b118-37389bd1a650 nodeName:}" failed. No retries permitted until 2026-09-14 11:00:47.918527343 +0000 UTC m=+5.981930238 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6547f534-40c9-42c7-b118-37389bd1a650-original-pull-secret") pod "global-pull-secret-syncer-96bxc" (UID: "6547f534-40c9-42c7-b118-37389bd1a650") : object "kube-system"/"original-pull-secret" not registered Sep 14 11:00:45.919077 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:45.918645 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 11:00:45.919077 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:45.918660 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 11:00:45.919077 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:45.918676 2556 projected.go:196] Error preparing data for projected volume kube-api-access-rnvm5 for pod openshift-network-diagnostics/network-check-target-d5x29: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:00:45.919077 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:45.918715 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6de42dc8-677e-4919-816e-19952a0ca273-kube-api-access-rnvm5 podName:6de42dc8-677e-4919-816e-19952a0ca273 nodeName:}" failed. No retries permitted until 2026-09-14 11:00:47.918701997 +0000 UTC m=+5.982104894 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rnvm5" (UniqueName: "kubernetes.io/projected/6de42dc8-677e-4919-816e-19952a0ca273-kube-api-access-rnvm5") pod "network-check-target-d5x29" (UID: "6de42dc8-677e-4919-816e-19952a0ca273") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:00:46.019366 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:46.019325 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs\") pod \"network-metrics-daemon-z75dj\" (UID: \"a69c5498-e86f-4c58-940f-1d14fa24a91a\") " pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:00:46.019561 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:46.019449 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:00:46.019561 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:46.019534 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs podName:a69c5498-e86f-4c58-940f-1d14fa24a91a nodeName:}" failed. No retries permitted until 2026-09-14 11:00:48.019512718 +0000 UTC m=+6.082915620 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs") pod "network-metrics-daemon-z75dj" (UID: "a69c5498-e86f-4c58-940f-1d14fa24a91a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:00:46.344569 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:46.344438 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 10:55:45 +0000 UTC" deadline="2028-05-01 02:29:33.323720524 +0000 UTC" Sep 14 11:00:46.344569 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:46.344482 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14271h28m46.97924383s" Sep 14 11:00:46.377126 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:46.377017 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:00:46.377296 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:46.377193 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z75dj" podUID="a69c5498-e86f-4c58-940f-1d14fa24a91a" Sep 14 11:00:46.377662 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:46.377641 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:00:46.377771 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:46.377749 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96bxc" podUID="6547f534-40c9-42c7-b118-37389bd1a650" Sep 14 11:00:46.377848 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:46.377828 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:00:46.377967 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:46.377909 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5x29" podUID="6de42dc8-677e-4919-816e-19952a0ca273" Sep 14 11:00:47.934463 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:47.934423 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6547f534-40c9-42c7-b118-37389bd1a650-original-pull-secret\") pod \"global-pull-secret-syncer-96bxc\" (UID: \"6547f534-40c9-42c7-b118-37389bd1a650\") " pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:00:47.935092 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:47.934497 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnvm5\" (UniqueName: \"kubernetes.io/projected/6de42dc8-677e-4919-816e-19952a0ca273-kube-api-access-rnvm5\") pod \"network-check-target-d5x29\" (UID: \"6de42dc8-677e-4919-816e-19952a0ca273\") " pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:00:47.935092 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:47.934649 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 11:00:47.935092 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:47.934672 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 11:00:47.935092 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:47.934685 2556 projected.go:196] Error preparing data for projected volume kube-api-access-rnvm5 for pod openshift-network-diagnostics/network-check-target-d5x29: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:00:47.935092 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:47.934754 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6de42dc8-677e-4919-816e-19952a0ca273-kube-api-access-rnvm5 podName:6de42dc8-677e-4919-816e-19952a0ca273 nodeName:}" failed. No retries permitted until 2026-09-14 11:00:51.934731024 +0000 UTC m=+9.998133934 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rnvm5" (UniqueName: "kubernetes.io/projected/6de42dc8-677e-4919-816e-19952a0ca273-kube-api-access-rnvm5") pod "network-check-target-d5x29" (UID: "6de42dc8-677e-4919-816e-19952a0ca273") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:00:47.935092 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:47.934827 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 11:00:47.935092 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:47.934864 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6547f534-40c9-42c7-b118-37389bd1a650-original-pull-secret podName:6547f534-40c9-42c7-b118-37389bd1a650 nodeName:}" failed. No retries permitted until 2026-09-14 11:00:51.934852246 +0000 UTC m=+9.998255137 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6547f534-40c9-42c7-b118-37389bd1a650-original-pull-secret") pod "global-pull-secret-syncer-96bxc" (UID: "6547f534-40c9-42c7-b118-37389bd1a650") : object "kube-system"/"original-pull-secret" not registered Sep 14 11:00:48.035482 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:48.035440 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs\") pod \"network-metrics-daemon-z75dj\" (UID: \"a69c5498-e86f-4c58-940f-1d14fa24a91a\") " pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:00:48.035689 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:48.035612 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:00:48.035689 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:48.035681 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs podName:a69c5498-e86f-4c58-940f-1d14fa24a91a nodeName:}" failed. No retries permitted until 2026-09-14 11:00:52.035663495 +0000 UTC m=+10.099066401 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs") pod "network-metrics-daemon-z75dj" (UID: "a69c5498-e86f-4c58-940f-1d14fa24a91a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:00:48.377298 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:48.377198 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:00:48.377483 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:48.377367 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5x29" podUID="6de42dc8-677e-4919-816e-19952a0ca273" Sep 14 11:00:48.377829 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:48.377801 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:00:48.377940 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:48.377924 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z75dj" podUID="a69c5498-e86f-4c58-940f-1d14fa24a91a" Sep 14 11:00:48.378004 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:48.377977 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:00:48.378072 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:48.378052 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96bxc" podUID="6547f534-40c9-42c7-b118-37389bd1a650" Sep 14 11:00:50.377822 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:50.377274 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:00:50.377822 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:50.377413 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5x29" podUID="6de42dc8-677e-4919-816e-19952a0ca273" Sep 14 11:00:50.377822 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:50.377494 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:00:50.377822 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:50.377621 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z75dj" podUID="a69c5498-e86f-4c58-940f-1d14fa24a91a" Sep 14 11:00:50.377822 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:50.377274 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:00:50.377822 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:50.377727 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96bxc" podUID="6547f534-40c9-42c7-b118-37389bd1a650" Sep 14 11:00:51.972148 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:51.971265 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnvm5\" (UniqueName: \"kubernetes.io/projected/6de42dc8-677e-4919-816e-19952a0ca273-kube-api-access-rnvm5\") pod \"network-check-target-d5x29\" (UID: \"6de42dc8-677e-4919-816e-19952a0ca273\") " pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:00:51.972148 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:51.971366 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6547f534-40c9-42c7-b118-37389bd1a650-original-pull-secret\") pod \"global-pull-secret-syncer-96bxc\" (UID: \"6547f534-40c9-42c7-b118-37389bd1a650\") " pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:00:51.972148 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:51.971505 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 11:00:51.972148 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:51.971580 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6547f534-40c9-42c7-b118-37389bd1a650-original-pull-secret podName:6547f534-40c9-42c7-b118-37389bd1a650 nodeName:}" failed. No retries permitted until 2026-09-14 11:00:59.971559625 +0000 UTC m=+18.034962542 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6547f534-40c9-42c7-b118-37389bd1a650-original-pull-secret") pod "global-pull-secret-syncer-96bxc" (UID: "6547f534-40c9-42c7-b118-37389bd1a650") : object "kube-system"/"original-pull-secret" not registered Sep 14 11:00:51.972148 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:51.972027 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 11:00:51.972148 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:51.972048 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 11:00:51.972148 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:51.972061 2556 projected.go:196] Error preparing data for projected volume kube-api-access-rnvm5 for pod openshift-network-diagnostics/network-check-target-d5x29: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:00:51.972148 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:51.972109 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6de42dc8-677e-4919-816e-19952a0ca273-kube-api-access-rnvm5 podName:6de42dc8-677e-4919-816e-19952a0ca273 nodeName:}" failed. No retries permitted until 2026-09-14 11:00:59.972093028 +0000 UTC m=+18.035495924 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rnvm5" (UniqueName: "kubernetes.io/projected/6de42dc8-677e-4919-816e-19952a0ca273-kube-api-access-rnvm5") pod "network-check-target-d5x29" (UID: "6de42dc8-677e-4919-816e-19952a0ca273") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:00:52.072572 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:52.072530 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs\") pod \"network-metrics-daemon-z75dj\" (UID: \"a69c5498-e86f-4c58-940f-1d14fa24a91a\") " pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:00:52.072775 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:52.072674 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:00:52.072775 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:52.072755 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs podName:a69c5498-e86f-4c58-940f-1d14fa24a91a nodeName:}" failed. No retries permitted until 2026-09-14 11:01:00.07273405 +0000 UTC m=+18.136136963 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs") pod "network-metrics-daemon-z75dj" (UID: "a69c5498-e86f-4c58-940f-1d14fa24a91a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:00:52.381294 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:52.378122 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:00:52.381294 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:52.378283 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5x29" podUID="6de42dc8-677e-4919-816e-19952a0ca273" Sep 14 11:00:52.381294 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:52.379853 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:00:52.381294 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:52.379956 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z75dj" podUID="a69c5498-e86f-4c58-940f-1d14fa24a91a" Sep 14 11:00:52.381294 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:52.380004 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:00:52.381294 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:52.380074 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96bxc" podUID="6547f534-40c9-42c7-b118-37389bd1a650" Sep 14 11:00:54.377110 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:54.376530 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:00:54.377110 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:54.376553 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:00:54.377110 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:54.376691 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z75dj" podUID="a69c5498-e86f-4c58-940f-1d14fa24a91a" Sep 14 11:00:54.377110 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:54.376738 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:00:54.377110 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:54.376870 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96bxc" podUID="6547f534-40c9-42c7-b118-37389bd1a650" Sep 14 11:00:54.377110 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:54.376959 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5x29" podUID="6de42dc8-677e-4919-816e-19952a0ca273" Sep 14 11:00:56.376855 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:56.376768 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:00:56.376855 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:56.376792 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:00:56.377384 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:56.376901 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z75dj" podUID="a69c5498-e86f-4c58-940f-1d14fa24a91a" Sep 14 11:00:56.377384 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:56.376952 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96bxc" podUID="6547f534-40c9-42c7-b118-37389bd1a650" Sep 14 11:00:56.377384 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:56.377004 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:00:56.377384 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:56.377082 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5x29" podUID="6de42dc8-677e-4919-816e-19952a0ca273" Sep 14 11:00:58.377142 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:58.377099 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:00:58.377610 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:58.377109 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:00:58.377610 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:58.377241 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z75dj" podUID="a69c5498-e86f-4c58-940f-1d14fa24a91a" Sep 14 11:00:58.377610 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:00:58.377115 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:00:58.377610 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:58.377301 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96bxc" podUID="6547f534-40c9-42c7-b118-37389bd1a650" Sep 14 11:00:58.377610 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:00:58.377374 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5x29" podUID="6de42dc8-677e-4919-816e-19952a0ca273" Sep 14 11:01:00.036643 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:00.036609 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnvm5\" (UniqueName: \"kubernetes.io/projected/6de42dc8-677e-4919-816e-19952a0ca273-kube-api-access-rnvm5\") pod \"network-check-target-d5x29\" (UID: \"6de42dc8-677e-4919-816e-19952a0ca273\") " pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:01:00.037122 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:00.036675 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6547f534-40c9-42c7-b118-37389bd1a650-original-pull-secret\") pod \"global-pull-secret-syncer-96bxc\" (UID: \"6547f534-40c9-42c7-b118-37389bd1a650\") " pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:01:00.037122 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:00.036790 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 11:01:00.037122 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:00.036799 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 11:01:00.037122 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:00.036829 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 11:01:00.037122 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:00.036843 2556 projected.go:196] Error preparing data for projected volume kube-api-access-rnvm5 for pod openshift-network-diagnostics/network-check-target-d5x29: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:01:00.037122 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:00.036857 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6547f534-40c9-42c7-b118-37389bd1a650-original-pull-secret podName:6547f534-40c9-42c7-b118-37389bd1a650 nodeName:}" failed. No retries permitted until 2026-09-14 11:01:16.036838133 +0000 UTC m=+34.100241025 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6547f534-40c9-42c7-b118-37389bd1a650-original-pull-secret") pod "global-pull-secret-syncer-96bxc" (UID: "6547f534-40c9-42c7-b118-37389bd1a650") : object "kube-system"/"original-pull-secret" not registered Sep 14 11:01:00.037122 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:00.036895 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6de42dc8-677e-4919-816e-19952a0ca273-kube-api-access-rnvm5 podName:6de42dc8-677e-4919-816e-19952a0ca273 nodeName:}" failed. No retries permitted until 2026-09-14 11:01:16.036879132 +0000 UTC m=+34.100282035 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rnvm5" (UniqueName: "kubernetes.io/projected/6de42dc8-677e-4919-816e-19952a0ca273-kube-api-access-rnvm5") pod "network-check-target-d5x29" (UID: "6de42dc8-677e-4919-816e-19952a0ca273") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:01:00.137298 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:00.137203 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs\") pod \"network-metrics-daemon-z75dj\" (UID: \"a69c5498-e86f-4c58-940f-1d14fa24a91a\") " pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:01:00.137473 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:00.137359 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:01:00.137473 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:00.137433 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs podName:a69c5498-e86f-4c58-940f-1d14fa24a91a nodeName:}" failed. No retries permitted until 2026-09-14 11:01:16.137416948 +0000 UTC m=+34.200819841 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs") pod "network-metrics-daemon-z75dj" (UID: "a69c5498-e86f-4c58-940f-1d14fa24a91a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:01:00.376861 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:00.376828 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:01:00.377041 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:00.376869 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:01:00.377041 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:00.376833 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:01:00.377041 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:00.376960 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96bxc" podUID="6547f534-40c9-42c7-b118-37389bd1a650" Sep 14 11:01:00.377236 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:00.377035 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5x29" podUID="6de42dc8-677e-4919-816e-19952a0ca273" Sep 14 11:01:00.377236 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:00.377152 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z75dj" podUID="a69c5498-e86f-4c58-940f-1d14fa24a91a" Sep 14 11:01:02.376853 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:02.376821 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:01:02.377302 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:02.376902 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:01:02.377302 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:02.376911 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96bxc" podUID="6547f534-40c9-42c7-b118-37389bd1a650" Sep 14 11:01:02.377302 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:02.377018 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5x29" podUID="6de42dc8-677e-4919-816e-19952a0ca273" Sep 14 11:01:02.377302 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:02.377054 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:01:02.377302 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:02.377178 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z75dj" podUID="a69c5498-e86f-4c58-940f-1d14fa24a91a" Sep 14 11:01:03.464883 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:03.464647 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k7qpb" event={"ID":"624da2af-49e4-4059-9c72-20d2688683f0","Type":"ContainerStarted","Data":"39e4ca619eb835c770cca0b8cf3326dcf0b6059c62deb0d5cb63a816bdb44a1f"} Sep 14 11:01:03.466841 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:03.466765 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dk98m" event={"ID":"eb75ff85-d0ae-49a6-9f12-70361ba65c10","Type":"ContainerStarted","Data":"ac6b4a212d49be1c1f2bd562a57cd699cbc93e745532b5c913b8ebde0403d56b"} Sep 14 11:01:03.469789 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:03.469741 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-52.ec2.internal" event={"ID":"c127f793063e49cd83ca3a678e448bb8","Type":"ContainerStarted","Data":"521342a863accd32ec83f50568a07fda22ad730257cc370f801c100ff866e127"} Sep 14 11:01:03.473241 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:03.473215 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" event={"ID":"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3","Type":"ContainerStarted","Data":"75b6018e717e84664436fc56aa01c79b4440a3bf579f7a91ef40967195a79a76"} Sep 14 11:01:03.489907 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:03.489340 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-k7qpb" podStartSLOduration=1.7967606809999999 podStartE2EDuration="20.48931972s" totalImagesPullingTime="18.692559039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:03.488882704 +0000 UTC m=+21.552285626" watchObservedRunningTime="2026-09-14 11:01:03.48931972 +0000 UTC m=+21.552722631" Sep 14 11:01:03.530330 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:03.530076 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-52.ec2.internal" podStartSLOduration=19.530053489 podStartE2EDuration="19.530053489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:03.508066241 +0000 UTC m=+21.571469153" watchObservedRunningTime="2026-09-14 11:01:03.530053489 +0000 UTC m=+21.593456401" Sep 14 11:01:03.531397 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:03.531353 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dk98m" podStartSLOduration=1.886971927 podStartE2EDuration="20.531339531s" totalImagesPullingTime="18.644367604s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:03.529897841 +0000 UTC m=+21.593300752" watchObservedRunningTime="2026-09-14 11:01:03.531339531 +0000 UTC m=+21.594742443" Sep 14 11:01:04.377121 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:04.377073 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:01:04.377285 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:04.377073 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:01:04.377285 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:04.377193 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z75dj" podUID="a69c5498-e86f-4c58-940f-1d14fa24a91a" Sep 14 11:01:04.377285 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:04.377077 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:01:04.377430 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:04.377350 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5x29" podUID="6de42dc8-677e-4919-816e-19952a0ca273" Sep 14 11:01:04.377430 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:04.377256 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96bxc" podUID="6547f534-40c9-42c7-b118-37389bd1a650" Sep 14 11:01:04.476940 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:04.476908 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vtv8p" event={"ID":"e19a55ba-0b57-429c-a111-7b199d668301","Type":"ContainerStarted","Data":"6ae5eac151cf12ac6cf76fabc44f2413399cd20935243489f860dfcc210ef336"} Sep 14 11:01:04.478425 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:04.478406 2556 generic.go:356] "Generic (PLEG): container finished" podID="38ba6e4ef478d7cc956f3c99e2c2415d" containerID="5b4a4f18b1e00ba54afb7020217c4a5a27159b0badbd1e31be871a32e5dd6e9d" exitCode=0 Sep 14 11:01:04.478511 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:04.478468 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal" event={"ID":"38ba6e4ef478d7cc956f3c99e2c2415d","Type":"ContainerDied","Data":"5b4a4f18b1e00ba54afb7020217c4a5a27159b0badbd1e31be871a32e5dd6e9d"} Sep 14 11:01:04.479962 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:04.479935 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mcxhz" event={"ID":"2f816499-fb81-4f59-bb3f-986f34d9c571","Type":"ContainerStarted","Data":"3340af544ff828d131e544a53e7bbae896b17c568b6d8efc59d0c3fb60d2bcf3"} Sep 14 11:01:04.481367 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:04.481346 2556 generic.go:356] "Generic (PLEG): container finished" podID="6133abcd-35d6-4ac5-8a38-e88d153bdaac" containerID="d54ebe4f56e8136302b458df20024786ee323a1dac748635befc6f876031c0a6" exitCode=0 Sep 14 11:01:04.481467 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:04.481423 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2md5p" event={"ID":"6133abcd-35d6-4ac5-8a38-e88d153bdaac","Type":"ContainerDied","Data":"d54ebe4f56e8136302b458df20024786ee323a1dac748635befc6f876031c0a6"} Sep 14 11:01:04.482989 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:04.482967 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" event={"ID":"4e67edae-3518-4742-b2cd-c3d3ed2762a2","Type":"ContainerStarted","Data":"f77e875e2d804f697332ef1ca2ad8e763c29cd2606a26f347e721a0cc76cdad2"} Sep 14 11:01:04.484497 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:04.484467 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5qhj6" event={"ID":"27fdd2fa-1378-4960-bd86-fd1dd84516c3","Type":"ContainerStarted","Data":"b52df4b09ece6107706de769a1b86b3bbb61ab66278b6de9219069f127ae0d72"} Sep 14 11:01:04.487280 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:04.487262 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" event={"ID":"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3","Type":"ContainerStarted","Data":"62f961d060f7f0d24a43d33a4f3184e16bebc64897e9fc99967decac3a5ba5e6"} Sep 14 11:01:04.487369 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:04.487315 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" event={"ID":"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3","Type":"ContainerStarted","Data":"4f86002297bf4e8963e4d5e1977a318a53ca0f281d9c679c864edfc0acafdd7e"} Sep 14 11:01:04.487369 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:04.487347 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" event={"ID":"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3","Type":"ContainerStarted","Data":"60089f5bbfd34254d76f25602fe320ee30f086095fd555fc6d74146fa5e83fa7"} Sep 14 11:01:04.487437 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:04.487380 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" event={"ID":"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3","Type":"ContainerStarted","Data":"577a97e974f30de5173c85071de758fdba479ecbf0a99084343aa9a86afab952"} Sep 14 11:01:04.487437 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:04.487411 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" event={"ID":"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3","Type":"ContainerStarted","Data":"9a2d138b18ce2302981bf31af023c6a9be2db83150e2a84d884abc2a64d347cf"} Sep 14 11:01:04.551355 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:04.551316 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vtv8p" podStartSLOduration=6.358589882 podStartE2EDuration="21.551301354s" totalImagesPullingTime="15.192711472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:04.510834032 +0000 UTC m=+22.574236944" watchObservedRunningTime="2026-09-14 11:01:04.551301354 +0000 UTC m=+22.614704265" Sep 14 11:01:04.709248 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:04.709036 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5qhj6" podStartSLOduration=6.551379493 podStartE2EDuration="21.709015255s" totalImagesPullingTime="15.157635762s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:04.635390896 +0000 UTC m=+22.698793808" watchObservedRunningTime="2026-09-14 11:01:04.709015255 +0000 UTC m=+22.772418147" Sep 14 11:01:04.709703 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:04.709677 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mcxhz" podStartSLOduration=3.115820068 podStartE2EDuration="21.70966965s" totalImagesPullingTime="18.593849582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:04.708986901 +0000 UTC m=+22.772389811" watchObservedRunningTime="2026-09-14 11:01:04.70966965 +0000 UTC m=+22.773072560" Sep 14 11:01:05.491209 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:05.491165 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7cn6h" event={"ID":"4259f994-0c08-408e-8db2-b3dc44beb77d","Type":"ContainerStarted","Data":"b4fd99656a3a0b1130cead3aed7fb50816b28802c2eb33978354378738d35fd7"} Sep 14 11:01:05.493135 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:05.493104 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal" event={"ID":"38ba6e4ef478d7cc956f3c99e2c2415d","Type":"ContainerStarted","Data":"a707f70f4a7edc711f234325e202050b5d1a72d78a25937438914973cf42d53e"} Sep 14 11:01:05.509390 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:05.509345 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7cn6h" podStartSLOduration=7.334925903 podStartE2EDuration="22.509329118s" totalImagesPullingTime="15.174403215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:05.509043966 +0000 UTC m=+23.572446877" watchObservedRunningTime="2026-09-14 11:01:05.509329118 +0000 UTC m=+23.572732033" Sep 14 11:01:05.525826 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:05.525784 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-52.ec2.internal" podStartSLOduration=21.525770846 podStartE2EDuration="21.525770846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:05.525541189 +0000 UTC m=+23.588944195" watchObservedRunningTime="2026-09-14 11:01:05.525770846 +0000 UTC m=+23.589173757" Sep 14 11:01:06.376768 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:06.376663 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:01:06.376936 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:06.376661 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:01:06.376936 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:06.376787 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5x29" podUID="6de42dc8-677e-4919-816e-19952a0ca273" Sep 14 11:01:06.376936 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:06.376870 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96bxc" podUID="6547f534-40c9-42c7-b118-37389bd1a650" Sep 14 11:01:06.376936 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:06.376663 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:01:06.377149 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:06.376967 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z75dj" podUID="a69c5498-e86f-4c58-940f-1d14fa24a91a" Sep 14 11:01:06.499912 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:06.499871 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" event={"ID":"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3","Type":"ContainerStarted","Data":"8c30cb1a1a8c0cd18fa9d78eae406f4c1fd4bb61bb602007cc24729a2e0871fb"} Sep 14 11:01:07.007029 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:07.006989 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vtv8p" Sep 14 11:01:07.007932 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:07.007914 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vtv8p" Sep 14 11:01:08.376861 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:08.376819 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:01:08.377407 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:08.376935 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5x29" podUID="6de42dc8-677e-4919-816e-19952a0ca273" Sep 14 11:01:08.377407 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:08.376817 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:01:08.377407 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:08.376989 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:01:08.377407 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:08.377080 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96bxc" podUID="6547f534-40c9-42c7-b118-37389bd1a650" Sep 14 11:01:08.377407 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:08.377221 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z75dj" podUID="a69c5498-e86f-4c58-940f-1d14fa24a91a" Sep 14 11:01:08.833993 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:08.833850 2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 14 11:01:09.358461 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:09.358222 2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-14T11:01:08.833867463Z","UUID":"30343933-9e24-4d50-b594-3d30452115ef","Handler":null,"Name":"","Endpoint":""} Sep 14 11:01:09.359629 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:09.359609 2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 14 11:01:09.359734 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:09.359643 2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 14 11:01:09.511478 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:09.511445 2556 generic.go:356] "Generic (PLEG): container finished" podID="6133abcd-35d6-4ac5-8a38-e88d153bdaac" containerID="63873d3b5a786b724377e3fa77510e30d9e99149c8ded3c8438fea38d5214878" exitCode=0 Sep 14 11:01:09.512253 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:09.511531 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2md5p" event={"ID":"6133abcd-35d6-4ac5-8a38-e88d153bdaac","Type":"ContainerDied","Data":"63873d3b5a786b724377e3fa77510e30d9e99149c8ded3c8438fea38d5214878"} Sep 14 11:01:09.513444 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:09.513424 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" event={"ID":"4e67edae-3518-4742-b2cd-c3d3ed2762a2","Type":"ContainerStarted","Data":"0550a0dedd3013c754fe50cb531b31b5a850e77c9983ea3cf2967939f9165795"} Sep 14 11:01:09.517117 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:09.517078 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" event={"ID":"4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3","Type":"ContainerStarted","Data":"4da89cf8b8d7d80a9e0ae345d6cdabd0cf28ae3d435134ae99266b789f7c5148"} Sep 14 11:01:09.517372 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:09.517355 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:01:09.532446 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:09.532426 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:01:09.573223 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:09.573178 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" podStartSLOduration=7.889030548 podStartE2EDuration="26.573163514s" totalImagesPullingTime="18.684132966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:09.571916355 +0000 UTC m=+27.635319266" watchObservedRunningTime="2026-09-14 11:01:09.573163514 +0000 UTC m=+27.636566424" Sep 14 11:01:10.380797 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:10.380770 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:01:10.380895 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:10.380771 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:01:10.380895 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:10.380876 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5x29" podUID="6de42dc8-677e-4919-816e-19952a0ca273" Sep 14 11:01:10.381019 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:10.380993 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z75dj" podUID="a69c5498-e86f-4c58-940f-1d14fa24a91a" Sep 14 11:01:10.381619 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:10.381602 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:01:10.381690 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:10.381674 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96bxc" podUID="6547f534-40c9-42c7-b118-37389bd1a650" Sep 14 11:01:10.521622 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:10.521584 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2md5p" event={"ID":"6133abcd-35d6-4ac5-8a38-e88d153bdaac","Type":"ContainerStarted","Data":"c1d0946515c3de8a39a480262d85fa0de49352f987a9ab164fbd0e694683da9d"} Sep 14 11:01:10.521622 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:10.521612 2556 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 14 11:01:10.522243 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:10.522182 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:01:10.536172 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:10.536152 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:01:10.802306 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:10.802220 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d5x29"] Sep 14 11:01:10.802465 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:10.802362 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:01:10.802530 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:10.802471 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5x29" podUID="6de42dc8-677e-4919-816e-19952a0ca273" Sep 14 11:01:10.810715 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:10.810683 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-96bxc"] Sep 14 11:01:10.810834 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:10.810781 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:01:10.810920 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:10.810875 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96bxc" podUID="6547f534-40c9-42c7-b118-37389bd1a650" Sep 14 11:01:10.811178 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:10.811155 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z75dj"] Sep 14 11:01:10.811262 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:10.811240 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:01:10.811376 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:10.811344 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z75dj" podUID="a69c5498-e86f-4c58-940f-1d14fa24a91a" Sep 14 11:01:11.526108 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:11.525831 2556 generic.go:356] "Generic (PLEG): container finished" podID="6133abcd-35d6-4ac5-8a38-e88d153bdaac" containerID="c1d0946515c3de8a39a480262d85fa0de49352f987a9ab164fbd0e694683da9d" exitCode=0 Sep 14 11:01:11.526108 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:11.525905 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2md5p" event={"ID":"6133abcd-35d6-4ac5-8a38-e88d153bdaac","Type":"ContainerDied","Data":"c1d0946515c3de8a39a480262d85fa0de49352f987a9ab164fbd0e694683da9d"} Sep 14 11:01:11.528392 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:11.528362 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" event={"ID":"4e67edae-3518-4742-b2cd-c3d3ed2762a2","Type":"ContainerStarted","Data":"4e42588e9c406f38a14af1de0362bcc4e55cd71b336f03691e0e56a3252081d0"} Sep 14 11:01:11.528463 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:11.528408 2556 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 14 11:01:11.571625 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:11.571572 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9l96" podStartSLOduration=2.589023124 podStartE2EDuration="28.571556432s" totalImagesPullingTime="25.982533308s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:11.571220672 +0000 UTC m=+29.634623596" watchObservedRunningTime="2026-09-14 11:01:11.571556432 +0000 UTC m=+29.634959335" Sep 14 11:01:12.377023 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:12.376924 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:01:12.377023 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:12.376996 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:01:12.377023 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:12.377002 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5x29" podUID="6de42dc8-677e-4919-816e-19952a0ca273" Sep 14 11:01:12.377265 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:12.377117 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z75dj" podUID="a69c5498-e86f-4c58-940f-1d14fa24a91a" Sep 14 11:01:12.377265 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:12.377138 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:01:12.377265 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:12.377233 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96bxc" podUID="6547f534-40c9-42c7-b118-37389bd1a650" Sep 14 11:01:12.533329 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:12.533297 2556 generic.go:356] "Generic (PLEG): container finished" podID="6133abcd-35d6-4ac5-8a38-e88d153bdaac" containerID="a7249215ca56da711b41d4c529bb068a377fd9280da78544e4bccbfaf2cbba7f" exitCode=0 Sep 14 11:01:12.533750 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:12.533386 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2md5p" event={"ID":"6133abcd-35d6-4ac5-8a38-e88d153bdaac","Type":"ContainerDied","Data":"a7249215ca56da711b41d4c529bb068a377fd9280da78544e4bccbfaf2cbba7f"} Sep 14 11:01:12.533750 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:12.533449 2556 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 14 11:01:12.696742 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:12.696664 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vtv8p" Sep 14 11:01:12.696907 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:12.696787 2556 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 14 11:01:12.697233 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:12.697217 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vtv8p" Sep 14 11:01:14.376621 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:14.376589 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:01:14.376621 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:14.376606 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:01:14.377196 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:14.376589 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:01:14.377196 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:14.376715 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96bxc" podUID="6547f534-40c9-42c7-b118-37389bd1a650" Sep 14 11:01:14.377196 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:14.376811 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z75dj" podUID="a69c5498-e86f-4c58-940f-1d14fa24a91a" Sep 14 11:01:14.377196 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:14.376915 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5x29" podUID="6de42dc8-677e-4919-816e-19952a0ca273" Sep 14 11:01:14.502945 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:14.502907 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:01:14.503222 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:14.503171 2556 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 14 11:01:14.518676 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:14.518627 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" podUID="4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3" containerName="ovnkube-controller" probeResult="failure" output="" Sep 14 11:01:14.528043 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:14.528013 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" podUID="4bf809e2-5d44-4a24-a5d9-58e6e49ab7a3" containerName="ovnkube-controller" probeResult="failure" output="" Sep 14 11:01:16.061479 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.061394 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnvm5\" (UniqueName: \"kubernetes.io/projected/6de42dc8-677e-4919-816e-19952a0ca273-kube-api-access-rnvm5\") pod \"network-check-target-d5x29\" (UID: \"6de42dc8-677e-4919-816e-19952a0ca273\") " pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:01:16.061892 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.061485 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6547f534-40c9-42c7-b118-37389bd1a650-original-pull-secret\") pod \"global-pull-secret-syncer-96bxc\" (UID: \"6547f534-40c9-42c7-b118-37389bd1a650\") " pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:01:16.061892 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:16.061550 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 11:01:16.061892 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:16.061575 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 11:01:16.061892 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:16.061588 2556 projected.go:196] Error preparing data for projected volume kube-api-access-rnvm5 for pod openshift-network-diagnostics/network-check-target-d5x29: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:01:16.061892 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:16.061599 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 11:01:16.061892 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:16.061648 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6de42dc8-677e-4919-816e-19952a0ca273-kube-api-access-rnvm5 podName:6de42dc8-677e-4919-816e-19952a0ca273 nodeName:}" failed. No retries permitted until 2026-09-14 11:01:48.061635247 +0000 UTC m=+66.125038136 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rnvm5" (UniqueName: "kubernetes.io/projected/6de42dc8-677e-4919-816e-19952a0ca273-kube-api-access-rnvm5") pod "network-check-target-d5x29" (UID: "6de42dc8-677e-4919-816e-19952a0ca273") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:01:16.061892 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:16.061664 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6547f534-40c9-42c7-b118-37389bd1a650-original-pull-secret podName:6547f534-40c9-42c7-b118-37389bd1a650 nodeName:}" failed. No retries permitted until 2026-09-14 11:01:48.061658537 +0000 UTC m=+66.125061425 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6547f534-40c9-42c7-b118-37389bd1a650-original-pull-secret") pod "global-pull-secret-syncer-96bxc" (UID: "6547f534-40c9-42c7-b118-37389bd1a650") : object "kube-system"/"original-pull-secret" not registered Sep 14 11:01:16.162723 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.162683 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs\") pod \"network-metrics-daemon-z75dj\" (UID: \"a69c5498-e86f-4c58-940f-1d14fa24a91a\") " pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:01:16.162901 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:16.162799 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:01:16.162901 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:16.162878 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs podName:a69c5498-e86f-4c58-940f-1d14fa24a91a nodeName:}" failed. No retries permitted until 2026-09-14 11:01:48.162859569 +0000 UTC m=+66.226262466 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs") pod "network-metrics-daemon-z75dj" (UID: "a69c5498-e86f-4c58-940f-1d14fa24a91a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:01:16.199007 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.198975 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-52.ec2.internal" event="NodeReady" Sep 14 11:01:16.199193 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.199142 2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 14 11:01:16.241653 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.241619 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55555f4b99-6hgh7"] Sep 14 11:01:16.270888 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.270855 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-77b554968f-rrgfr"] Sep 14 11:01:16.271130 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.271042 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:16.274702 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.274574 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 14 11:01:16.275306 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.275286 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 14 11:01:16.275657 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.275637 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 14 11:01:16.275753 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.275715 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nd2nr\"" Sep 14 11:01:16.286480 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.286461 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 14 11:01:16.286579 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.286519 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9"] Sep 14 11:01:16.286686 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.286668 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77b554968f-rrgfr" Sep 14 11:01:16.295192 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.295145 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 14 11:01:16.295516 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.295497 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 14 11:01:16.295516 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.295497 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 14 11:01:16.295671 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.295595 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 14 11:01:16.306517 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.306490 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cd76b959d-4hcwn"] Sep 14 11:01:16.306616 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.306548 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" Sep 14 11:01:16.317271 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.317252 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 14 11:01:16.317762 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.317745 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 14 11:01:16.317848 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.317813 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 14 11:01:16.319036 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.319020 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 14 11:01:16.324773 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.324750 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9"] Sep 14 11:01:16.324861 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.324787 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-77b554968f-rrgfr"] Sep 14 11:01:16.324861 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.324818 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cd76b959d-4hcwn"] Sep 14 11:01:16.324861 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.324836 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-k264k"] Sep 14 11:01:16.325002 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.324937 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cd76b959d-4hcwn" Sep 14 11:01:16.329523 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.329504 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 14 11:01:16.330135 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.330116 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-jn2qr\"" Sep 14 11:01:16.336836 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.336815 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55555f4b99-6hgh7"] Sep 14 11:01:16.336927 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.336856 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nvm9v"] Sep 14 11:01:16.336976 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.336926 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k264k" Sep 14 11:01:16.350364 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.350338 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zpbn2\"" Sep 14 11:01:16.350590 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.350566 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 14 11:01:16.351206 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.351185 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 14 11:01:16.354916 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.354898 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nvm9v"] Sep 14 11:01:16.355021 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.354928 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k264k"] Sep 14 11:01:16.355065 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.355029 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nvm9v" Sep 14 11:01:16.359046 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.359027 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 14 11:01:16.359404 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.359383 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 14 11:01:16.359475 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.359438 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qzhb6\"" Sep 14 11:01:16.359576 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.359560 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 14 11:01:16.364501 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.364482 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-certificates\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:16.364585 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.364509 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-ca-trust-extracted\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:16.364585 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.364533 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-trusted-ca\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:16.364585 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.364555 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk9s8\" (UniqueName: \"kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-kube-api-access-kk9s8\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:16.364694 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.364643 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:16.364694 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.364672 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-image-registry-private-configuration\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:16.364769 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.364693 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-bound-sa-token\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:16.364769 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.364713 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-installation-pull-secrets\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:16.376619 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.376597 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:01:16.376619 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.376614 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:01:16.376804 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.376786 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:01:16.379485 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.379468 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 14 11:01:16.379575 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.379515 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 14 11:01:16.379865 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.379848 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 14 11:01:16.379865 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.379858 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mckdx\"" Sep 14 11:01:16.380111 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.380080 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-npv4p\"" Sep 14 11:01:16.380206 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.380165 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 14 11:01:16.465944 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.465912 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/750c7567-a733-462b-82ba-00c9e6c0065e-hub\") pod \"cluster-proxy-proxy-agent-7df45887fb-r4dw9\" (UID: \"750c7567-a733-462b-82ba-00c9e6c0065e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" Sep 14 11:01:16.465944 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.465944 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/750c7567-a733-462b-82ba-00c9e6c0065e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7df45887fb-r4dw9\" (UID: \"750c7567-a733-462b-82ba-00c9e6c0065e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" Sep 14 11:01:16.466158 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.465979 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-certificates\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:16.466158 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.465996 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc2hp\" (UniqueName: \"kubernetes.io/projected/750c7567-a733-462b-82ba-00c9e6c0065e-kube-api-access-gc2hp\") pod \"cluster-proxy-proxy-agent-7df45887fb-r4dw9\" (UID: \"750c7567-a733-462b-82ba-00c9e6c0065e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" Sep 14 11:01:16.466158 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.466058 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-ca-trust-extracted\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:16.466158 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.466106 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bd76a15-e9b6-402e-ad38-e1a3ab4a1a18-cert\") pod \"ingress-canary-nvm9v\" (UID: \"9bd76a15-e9b6-402e-ad38-e1a3ab4a1a18\") " pod="openshift-ingress-canary/ingress-canary-nvm9v" Sep 14 11:01:16.466158 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.466130 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-metrics-tls\") pod \"dns-default-k264k\" (UID: \"e59abbc8-c9b6-45ec-ae03-a7b534ca02df\") " pod="openshift-dns/dns-default-k264k" Sep 14 11:01:16.466158 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.466150 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-tmp-dir\") pod \"dns-default-k264k\" (UID: \"e59abbc8-c9b6-45ec-ae03-a7b534ca02df\") " pod="openshift-dns/dns-default-k264k" Sep 14 11:01:16.466450 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.466218 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28232384-152c-4039-bbfa-bd76716f9e60-tmp\") pod \"klusterlet-addon-workmgr-77b554968f-rrgfr\" (UID: \"28232384-152c-4039-bbfa-bd76716f9e60\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77b554968f-rrgfr" Sep 14 11:01:16.466450 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.466262 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-trusted-ca\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:16.466450 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.466291 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww46l\" (UniqueName: \"kubernetes.io/projected/54f80caa-892f-4553-88ba-018c70307b68-kube-api-access-ww46l\") pod \"managed-serviceaccount-addon-agent-7cd76b959d-4hcwn\" (UID: \"54f80caa-892f-4553-88ba-018c70307b68\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cd76b959d-4hcwn" Sep 14 11:01:16.466450 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.466333 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/750c7567-a733-462b-82ba-00c9e6c0065e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7df45887fb-r4dw9\" (UID: \"750c7567-a733-462b-82ba-00c9e6c0065e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" Sep 14 11:01:16.466450 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.466361 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/28232384-152c-4039-bbfa-bd76716f9e60-klusterlet-config\") pod \"klusterlet-addon-workmgr-77b554968f-rrgfr\" (UID: \"28232384-152c-4039-bbfa-bd76716f9e60\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77b554968f-rrgfr" Sep 14 11:01:16.466450 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.466379 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-ca-trust-extracted\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:16.466450 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.466385 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4882q\" (UniqueName: \"kubernetes.io/projected/28232384-152c-4039-bbfa-bd76716f9e60-kube-api-access-4882q\") pod \"klusterlet-addon-workmgr-77b554968f-rrgfr\" (UID: \"28232384-152c-4039-bbfa-bd76716f9e60\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77b554968f-rrgfr" Sep 14 11:01:16.466450 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.466431 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk9s8\" (UniqueName: \"kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-kube-api-access-kk9s8\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:16.466791 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.466487 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/750c7567-a733-462b-82ba-00c9e6c0065e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7df45887fb-r4dw9\" (UID: \"750c7567-a733-462b-82ba-00c9e6c0065e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" Sep 14 11:01:16.466791 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.466515 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54f80caa-892f-4553-88ba-018c70307b68-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7cd76b959d-4hcwn\" (UID: \"54f80caa-892f-4553-88ba-018c70307b68\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cd76b959d-4hcwn" Sep 14 11:01:16.466791 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.466580 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:16.466791 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.466579 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-certificates\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:16.466791 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.466622 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-image-registry-private-configuration\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:16.466791 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.466659 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-bound-sa-token\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:16.466791 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:16.466674 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 11:01:16.466791 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.466685 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/750c7567-a733-462b-82ba-00c9e6c0065e-ca\") pod \"cluster-proxy-proxy-agent-7df45887fb-r4dw9\" (UID: \"750c7567-a733-462b-82ba-00c9e6c0065e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" Sep 14 11:01:16.466791 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:16.466690 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55555f4b99-6hgh7: secret "image-registry-tls" not found Sep 14 11:01:16.466791 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.466710 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-config-volume\") pod \"dns-default-k264k\" (UID: \"e59abbc8-c9b6-45ec-ae03-a7b534ca02df\") " pod="openshift-dns/dns-default-k264k" Sep 14 11:01:16.466791 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.466734 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvgtp\" (UniqueName: \"kubernetes.io/projected/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-kube-api-access-dvgtp\") pod \"dns-default-k264k\" (UID: \"e59abbc8-c9b6-45ec-ae03-a7b534ca02df\") " pod="openshift-dns/dns-default-k264k" Sep 14 11:01:16.466791 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:16.466755 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls podName:f71e3f64-7c46-4b5f-8e66-8b20d9d32983 nodeName:}" failed. No retries permitted until 2026-09-14 11:01:16.966735651 +0000 UTC m=+35.030138545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls") pod "image-registry-55555f4b99-6hgh7" (UID: "f71e3f64-7c46-4b5f-8e66-8b20d9d32983") : secret "image-registry-tls" not found Sep 14 11:01:16.467294 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.466805 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-installation-pull-secrets\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:16.467294 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.466827 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7pvv\" (UniqueName: \"kubernetes.io/projected/9bd76a15-e9b6-402e-ad38-e1a3ab4a1a18-kube-api-access-d7pvv\") pod \"ingress-canary-nvm9v\" (UID: \"9bd76a15-e9b6-402e-ad38-e1a3ab4a1a18\") " pod="openshift-ingress-canary/ingress-canary-nvm9v" Sep 14 11:01:16.467358 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.467306 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-trusted-ca\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:16.470576 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.470556 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-installation-pull-secrets\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:16.470688 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.470567 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-image-registry-private-configuration\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:16.481936 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.481912 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk9s8\" (UniqueName: \"kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-kube-api-access-kk9s8\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:16.482813 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.482798 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-bound-sa-token\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:16.567580 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.567519 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/750c7567-a733-462b-82ba-00c9e6c0065e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7df45887fb-r4dw9\" (UID: \"750c7567-a733-462b-82ba-00c9e6c0065e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" Sep 14 11:01:16.567580 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.567554 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/28232384-152c-4039-bbfa-bd76716f9e60-klusterlet-config\") pod \"klusterlet-addon-workmgr-77b554968f-rrgfr\" (UID: \"28232384-152c-4039-bbfa-bd76716f9e60\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77b554968f-rrgfr" Sep 14 11:01:16.567580 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.567571 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4882q\" (UniqueName: \"kubernetes.io/projected/28232384-152c-4039-bbfa-bd76716f9e60-kube-api-access-4882q\") pod \"klusterlet-addon-workmgr-77b554968f-rrgfr\" (UID: \"28232384-152c-4039-bbfa-bd76716f9e60\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77b554968f-rrgfr" Sep 14 11:01:16.567809 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.567586 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/750c7567-a733-462b-82ba-00c9e6c0065e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7df45887fb-r4dw9\" (UID: \"750c7567-a733-462b-82ba-00c9e6c0065e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" Sep 14 11:01:16.567809 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.567605 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54f80caa-892f-4553-88ba-018c70307b68-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7cd76b959d-4hcwn\" (UID: \"54f80caa-892f-4553-88ba-018c70307b68\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cd76b959d-4hcwn" Sep 14 11:01:16.567809 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.567799 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/750c7567-a733-462b-82ba-00c9e6c0065e-ca\") pod \"cluster-proxy-proxy-agent-7df45887fb-r4dw9\" (UID: \"750c7567-a733-462b-82ba-00c9e6c0065e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" Sep 14 11:01:16.567954 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.567830 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-config-volume\") pod \"dns-default-k264k\" (UID: \"e59abbc8-c9b6-45ec-ae03-a7b534ca02df\") " pod="openshift-dns/dns-default-k264k" Sep 14 11:01:16.567954 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.567854 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvgtp\" (UniqueName: \"kubernetes.io/projected/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-kube-api-access-dvgtp\") pod \"dns-default-k264k\" (UID: \"e59abbc8-c9b6-45ec-ae03-a7b534ca02df\") " pod="openshift-dns/dns-default-k264k" Sep 14 11:01:16.567954 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.567887 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7pvv\" (UniqueName: \"kubernetes.io/projected/9bd76a15-e9b6-402e-ad38-e1a3ab4a1a18-kube-api-access-d7pvv\") pod \"ingress-canary-nvm9v\" (UID: \"9bd76a15-e9b6-402e-ad38-e1a3ab4a1a18\") " pod="openshift-ingress-canary/ingress-canary-nvm9v" Sep 14 11:01:16.567954 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.567933 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/750c7567-a733-462b-82ba-00c9e6c0065e-hub\") pod \"cluster-proxy-proxy-agent-7df45887fb-r4dw9\" (UID: \"750c7567-a733-462b-82ba-00c9e6c0065e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" Sep 14 11:01:16.568180 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.567961 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/750c7567-a733-462b-82ba-00c9e6c0065e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7df45887fb-r4dw9\" (UID: \"750c7567-a733-462b-82ba-00c9e6c0065e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" Sep 14 11:01:16.568180 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.568012 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc2hp\" (UniqueName: \"kubernetes.io/projected/750c7567-a733-462b-82ba-00c9e6c0065e-kube-api-access-gc2hp\") pod \"cluster-proxy-proxy-agent-7df45887fb-r4dw9\" (UID: \"750c7567-a733-462b-82ba-00c9e6c0065e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" Sep 14 11:01:16.568180 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.568046 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bd76a15-e9b6-402e-ad38-e1a3ab4a1a18-cert\") pod \"ingress-canary-nvm9v\" (UID: \"9bd76a15-e9b6-402e-ad38-e1a3ab4a1a18\") " pod="openshift-ingress-canary/ingress-canary-nvm9v" Sep 14 11:01:16.568180 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.568069 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-metrics-tls\") pod \"dns-default-k264k\" (UID: \"e59abbc8-c9b6-45ec-ae03-a7b534ca02df\") " pod="openshift-dns/dns-default-k264k" Sep 14 11:01:16.568180 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.568109 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-tmp-dir\") pod \"dns-default-k264k\" (UID: \"e59abbc8-c9b6-45ec-ae03-a7b534ca02df\") " pod="openshift-dns/dns-default-k264k" Sep 14 11:01:16.568180 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.568143 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28232384-152c-4039-bbfa-bd76716f9e60-tmp\") pod \"klusterlet-addon-workmgr-77b554968f-rrgfr\" (UID: \"28232384-152c-4039-bbfa-bd76716f9e60\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77b554968f-rrgfr" Sep 14 11:01:16.568180 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.568177 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww46l\" (UniqueName: \"kubernetes.io/projected/54f80caa-892f-4553-88ba-018c70307b68-kube-api-access-ww46l\") pod \"managed-serviceaccount-addon-agent-7cd76b959d-4hcwn\" (UID: \"54f80caa-892f-4553-88ba-018c70307b68\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cd76b959d-4hcwn" Sep 14 11:01:16.568512 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.568444 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-config-volume\") pod \"dns-default-k264k\" (UID: \"e59abbc8-c9b6-45ec-ae03-a7b534ca02df\") " pod="openshift-dns/dns-default-k264k" Sep 14 11:01:16.568745 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.568713 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-tmp-dir\") pod \"dns-default-k264k\" (UID: \"e59abbc8-c9b6-45ec-ae03-a7b534ca02df\") " pod="openshift-dns/dns-default-k264k" Sep 14 11:01:16.569239 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:16.569217 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 11:01:16.569337 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:16.569308 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-metrics-tls podName:e59abbc8-c9b6-45ec-ae03-a7b534ca02df nodeName:}" failed. No retries permitted until 2026-09-14 11:01:17.069275705 +0000 UTC m=+35.132678610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-metrics-tls") pod "dns-default-k264k" (UID: "e59abbc8-c9b6-45ec-ae03-a7b534ca02df") : secret "dns-default-metrics-tls" not found Sep 14 11:01:16.569490 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.569465 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28232384-152c-4039-bbfa-bd76716f9e60-tmp\") pod \"klusterlet-addon-workmgr-77b554968f-rrgfr\" (UID: \"28232384-152c-4039-bbfa-bd76716f9e60\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77b554968f-rrgfr" Sep 14 11:01:16.570586 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.570553 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54f80caa-892f-4553-88ba-018c70307b68-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7cd76b959d-4hcwn\" (UID: \"54f80caa-892f-4553-88ba-018c70307b68\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cd76b959d-4hcwn" Sep 14 11:01:16.570710 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.570693 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/750c7567-a733-462b-82ba-00c9e6c0065e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7df45887fb-r4dw9\" (UID: \"750c7567-a733-462b-82ba-00c9e6c0065e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" Sep 14 11:01:16.570864 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.570842 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/750c7567-a733-462b-82ba-00c9e6c0065e-ca\") pod \"cluster-proxy-proxy-agent-7df45887fb-r4dw9\" (UID: \"750c7567-a733-462b-82ba-00c9e6c0065e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" Sep 14 11:01:16.571304 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.571274 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bd76a15-e9b6-402e-ad38-e1a3ab4a1a18-cert\") pod \"ingress-canary-nvm9v\" (UID: \"9bd76a15-e9b6-402e-ad38-e1a3ab4a1a18\") " pod="openshift-ingress-canary/ingress-canary-nvm9v" Sep 14 11:01:16.571304 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.571293 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/28232384-152c-4039-bbfa-bd76716f9e60-klusterlet-config\") pod \"klusterlet-addon-workmgr-77b554968f-rrgfr\" (UID: \"28232384-152c-4039-bbfa-bd76716f9e60\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77b554968f-rrgfr" Sep 14 11:01:16.571443 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.571410 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/750c7567-a733-462b-82ba-00c9e6c0065e-hub\") pod \"cluster-proxy-proxy-agent-7df45887fb-r4dw9\" (UID: \"750c7567-a733-462b-82ba-00c9e6c0065e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" Sep 14 11:01:16.571704 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.571682 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/750c7567-a733-462b-82ba-00c9e6c0065e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7df45887fb-r4dw9\" (UID: \"750c7567-a733-462b-82ba-00c9e6c0065e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" Sep 14 11:01:16.577005 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.576982 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/750c7567-a733-462b-82ba-00c9e6c0065e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7df45887fb-r4dw9\" (UID: \"750c7567-a733-462b-82ba-00c9e6c0065e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" Sep 14 11:01:16.586701 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.586679 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4882q\" (UniqueName: \"kubernetes.io/projected/28232384-152c-4039-bbfa-bd76716f9e60-kube-api-access-4882q\") pod \"klusterlet-addon-workmgr-77b554968f-rrgfr\" (UID: \"28232384-152c-4039-bbfa-bd76716f9e60\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77b554968f-rrgfr" Sep 14 11:01:16.595136 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.595115 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww46l\" (UniqueName: \"kubernetes.io/projected/54f80caa-892f-4553-88ba-018c70307b68-kube-api-access-ww46l\") pod \"managed-serviceaccount-addon-agent-7cd76b959d-4hcwn\" (UID: \"54f80caa-892f-4553-88ba-018c70307b68\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cd76b959d-4hcwn" Sep 14 11:01:16.595451 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.595432 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7pvv\" (UniqueName: \"kubernetes.io/projected/9bd76a15-e9b6-402e-ad38-e1a3ab4a1a18-kube-api-access-d7pvv\") pod \"ingress-canary-nvm9v\" (UID: \"9bd76a15-e9b6-402e-ad38-e1a3ab4a1a18\") " pod="openshift-ingress-canary/ingress-canary-nvm9v" Sep 14 11:01:16.595522 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.595452 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvgtp\" (UniqueName: \"kubernetes.io/projected/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-kube-api-access-dvgtp\") pod \"dns-default-k264k\" (UID: \"e59abbc8-c9b6-45ec-ae03-a7b534ca02df\") " pod="openshift-dns/dns-default-k264k" Sep 14 11:01:16.596023 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.596007 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc2hp\" (UniqueName: \"kubernetes.io/projected/750c7567-a733-462b-82ba-00c9e6c0065e-kube-api-access-gc2hp\") pod \"cluster-proxy-proxy-agent-7df45887fb-r4dw9\" (UID: \"750c7567-a733-462b-82ba-00c9e6c0065e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" Sep 14 11:01:16.596115 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.596082 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77b554968f-rrgfr" Sep 14 11:01:16.631189 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.631165 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" Sep 14 11:01:16.641261 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.641241 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cd76b959d-4hcwn" Sep 14 11:01:16.665171 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.665148 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nvm9v" Sep 14 11:01:16.909342 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.909165 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-77b554968f-rrgfr"] Sep 14 11:01:16.972318 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:16.972287 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:16.972428 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:16.972398 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 11:01:16.972428 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:16.972409 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55555f4b99-6hgh7: secret "image-registry-tls" not found Sep 14 11:01:16.972493 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:16.972457 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls podName:f71e3f64-7c46-4b5f-8e66-8b20d9d32983 nodeName:}" failed. No retries permitted until 2026-09-14 11:01:17.972444733 +0000 UTC m=+36.035847622 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls") pod "image-registry-55555f4b99-6hgh7" (UID: "f71e3f64-7c46-4b5f-8e66-8b20d9d32983") : secret "image-registry-tls" not found Sep 14 11:01:17.043107 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:17.043050 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cd76b959d-4hcwn"] Sep 14 11:01:17.045104 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:17.045016 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nvm9v"] Sep 14 11:01:17.045482 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:17.045464 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9"] Sep 14 11:01:17.048979 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:01:17.048951 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28232384_152c_4039_bbfa_bd76716f9e60.slice/crio-911152f825398d55f3dabbc6ada5e6ffc93355454a5c5334025c0addc5f3bfea WatchSource:0}: Error finding container 911152f825398d55f3dabbc6ada5e6ffc93355454a5c5334025c0addc5f3bfea: Status 404 returned error can't find the container with id 911152f825398d55f3dabbc6ada5e6ffc93355454a5c5334025c0addc5f3bfea Sep 14 11:01:17.049338 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:01:17.049313 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod750c7567_a733_462b_82ba_00c9e6c0065e.slice/crio-bf0f36544575d2fbb444f2088d8159821ffb1d8a332ba8669d2258297065bd3d WatchSource:0}: Error finding container bf0f36544575d2fbb444f2088d8159821ffb1d8a332ba8669d2258297065bd3d: Status 404 returned error can't find the container with id bf0f36544575d2fbb444f2088d8159821ffb1d8a332ba8669d2258297065bd3d Sep 14 11:01:17.050026 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:01:17.050003 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54f80caa_892f_4553_88ba_018c70307b68.slice/crio-457f9e2e4a2047e3a209b04192ad596265d21d1572b1a6b3331afdf3c063de37 WatchSource:0}: Error finding container 457f9e2e4a2047e3a209b04192ad596265d21d1572b1a6b3331afdf3c063de37: Status 404 returned error can't find the container with id 457f9e2e4a2047e3a209b04192ad596265d21d1572b1a6b3331afdf3c063de37 Sep 14 11:01:17.051022 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:01:17.050971 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd76a15_e9b6_402e_ad38_e1a3ab4a1a18.slice/crio-afa6e6b63dae4b7b114bcdb0b185cdef4a7e027cafd1bbba38c6fb32706310df WatchSource:0}: Error finding container afa6e6b63dae4b7b114bcdb0b185cdef4a7e027cafd1bbba38c6fb32706310df: Status 404 returned error can't find the container with id afa6e6b63dae4b7b114bcdb0b185cdef4a7e027cafd1bbba38c6fb32706310df Sep 14 11:01:17.073019 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:17.072987 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-metrics-tls\") pod \"dns-default-k264k\" (UID: \"e59abbc8-c9b6-45ec-ae03-a7b534ca02df\") " pod="openshift-dns/dns-default-k264k" Sep 14 11:01:17.073514 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:17.073147 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 11:01:17.073514 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:17.073210 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-metrics-tls podName:e59abbc8-c9b6-45ec-ae03-a7b534ca02df nodeName:}" failed. No retries permitted until 2026-09-14 11:01:18.073194297 +0000 UTC m=+36.136597186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-metrics-tls") pod "dns-default-k264k" (UID: "e59abbc8-c9b6-45ec-ae03-a7b534ca02df") : secret "dns-default-metrics-tls" not found Sep 14 11:01:17.546497 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:17.546418 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" event={"ID":"750c7567-a733-462b-82ba-00c9e6c0065e","Type":"ContainerStarted","Data":"bf0f36544575d2fbb444f2088d8159821ffb1d8a332ba8669d2258297065bd3d"} Sep 14 11:01:17.549193 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:17.549171 2556 generic.go:356] "Generic (PLEG): container finished" podID="6133abcd-35d6-4ac5-8a38-e88d153bdaac" containerID="b4d3cf701ce696f032e88da368254a4384b6d6d79fa2a0b4b2238c596bcf0b7b" exitCode=0 Sep 14 11:01:17.549315 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:17.549233 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2md5p" event={"ID":"6133abcd-35d6-4ac5-8a38-e88d153bdaac","Type":"ContainerDied","Data":"b4d3cf701ce696f032e88da368254a4384b6d6d79fa2a0b4b2238c596bcf0b7b"} Sep 14 11:01:17.550374 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:17.550354 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nvm9v" event={"ID":"9bd76a15-e9b6-402e-ad38-e1a3ab4a1a18","Type":"ContainerStarted","Data":"afa6e6b63dae4b7b114bcdb0b185cdef4a7e027cafd1bbba38c6fb32706310df"} Sep 14 11:01:17.551469 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:17.551442 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cd76b959d-4hcwn" event={"ID":"54f80caa-892f-4553-88ba-018c70307b68","Type":"ContainerStarted","Data":"457f9e2e4a2047e3a209b04192ad596265d21d1572b1a6b3331afdf3c063de37"} Sep 14 11:01:17.552600 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:17.552561 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77b554968f-rrgfr" event={"ID":"28232384-152c-4039-bbfa-bd76716f9e60","Type":"ContainerStarted","Data":"911152f825398d55f3dabbc6ada5e6ffc93355454a5c5334025c0addc5f3bfea"} Sep 14 11:01:17.979955 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:17.979866 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:17.980146 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:17.980082 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 11:01:17.980146 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:17.980115 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55555f4b99-6hgh7: secret "image-registry-tls" not found Sep 14 11:01:17.980265 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:17.980173 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls podName:f71e3f64-7c46-4b5f-8e66-8b20d9d32983 nodeName:}" failed. No retries permitted until 2026-09-14 11:01:19.980155481 +0000 UTC m=+38.043558371 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls") pod "image-registry-55555f4b99-6hgh7" (UID: "f71e3f64-7c46-4b5f-8e66-8b20d9d32983") : secret "image-registry-tls" not found Sep 14 11:01:18.080762 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:18.080722 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-metrics-tls\") pod \"dns-default-k264k\" (UID: \"e59abbc8-c9b6-45ec-ae03-a7b534ca02df\") " pod="openshift-dns/dns-default-k264k" Sep 14 11:01:18.081372 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:18.080892 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 11:01:18.081372 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:18.080953 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-metrics-tls podName:e59abbc8-c9b6-45ec-ae03-a7b534ca02df nodeName:}" failed. No retries permitted until 2026-09-14 11:01:20.080935568 +0000 UTC m=+38.144338458 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-metrics-tls") pod "dns-default-k264k" (UID: "e59abbc8-c9b6-45ec-ae03-a7b534ca02df") : secret "dns-default-metrics-tls" not found Sep 14 11:01:18.566135 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:18.566079 2556 generic.go:356] "Generic (PLEG): container finished" podID="6133abcd-35d6-4ac5-8a38-e88d153bdaac" containerID="d0fa7448697bbbf709ad49844ba2feb41af21171306076734509e444ee4b71d0" exitCode=0 Sep 14 11:01:18.566332 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:18.566172 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2md5p" event={"ID":"6133abcd-35d6-4ac5-8a38-e88d153bdaac","Type":"ContainerDied","Data":"d0fa7448697bbbf709ad49844ba2feb41af21171306076734509e444ee4b71d0"} Sep 14 11:01:19.577502 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:19.577460 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2md5p" event={"ID":"6133abcd-35d6-4ac5-8a38-e88d153bdaac","Type":"ContainerStarted","Data":"b801a575275fc1af128e418ea77a2075cd31c978fd1c9b783bc5d5e270cfca20"} Sep 14 11:01:19.996444 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:19.995720 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:19.996444 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:19.995903 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 11:01:19.996444 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:19.995921 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55555f4b99-6hgh7: secret "image-registry-tls" not found Sep 14 11:01:19.996444 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:19.995977 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls podName:f71e3f64-7c46-4b5f-8e66-8b20d9d32983 nodeName:}" failed. No retries permitted until 2026-09-14 11:01:23.995959762 +0000 UTC m=+42.059362657 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls") pod "image-registry-55555f4b99-6hgh7" (UID: "f71e3f64-7c46-4b5f-8e66-8b20d9d32983") : secret "image-registry-tls" not found Sep 14 11:01:20.096488 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:20.096451 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-metrics-tls\") pod \"dns-default-k264k\" (UID: \"e59abbc8-c9b6-45ec-ae03-a7b534ca02df\") " pod="openshift-dns/dns-default-k264k" Sep 14 11:01:20.096719 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:20.096699 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 11:01:20.096788 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:20.096769 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-metrics-tls podName:e59abbc8-c9b6-45ec-ae03-a7b534ca02df nodeName:}" failed. No retries permitted until 2026-09-14 11:01:24.09674935 +0000 UTC m=+42.160152245 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-metrics-tls") pod "dns-default-k264k" (UID: "e59abbc8-c9b6-45ec-ae03-a7b534ca02df") : secret "dns-default-metrics-tls" not found Sep 14 11:01:24.025486 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:24.025250 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:24.026037 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:24.025386 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 11:01:24.026037 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:24.025567 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55555f4b99-6hgh7: secret "image-registry-tls" not found Sep 14 11:01:24.026037 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:24.025659 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls podName:f71e3f64-7c46-4b5f-8e66-8b20d9d32983 nodeName:}" failed. No retries permitted until 2026-09-14 11:01:32.025638131 +0000 UTC m=+50.089041059 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls") pod "image-registry-55555f4b99-6hgh7" (UID: "f71e3f64-7c46-4b5f-8e66-8b20d9d32983") : secret "image-registry-tls" not found Sep 14 11:01:24.126047 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:24.126009 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-metrics-tls\") pod \"dns-default-k264k\" (UID: \"e59abbc8-c9b6-45ec-ae03-a7b534ca02df\") " pod="openshift-dns/dns-default-k264k" Sep 14 11:01:24.126221 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:24.126162 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 11:01:24.126267 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:24.126224 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-metrics-tls podName:e59abbc8-c9b6-45ec-ae03-a7b534ca02df nodeName:}" failed. No retries permitted until 2026-09-14 11:01:32.126209173 +0000 UTC m=+50.189612061 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-metrics-tls") pod "dns-default-k264k" (UID: "e59abbc8-c9b6-45ec-ae03-a7b534ca02df") : secret "dns-default-metrics-tls" not found Sep 14 11:01:25.597350 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:25.597315 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77b554968f-rrgfr" event={"ID":"28232384-152c-4039-bbfa-bd76716f9e60","Type":"ContainerStarted","Data":"fbf5245808d3ba2f192a4ef2de963625c9684fc81064fd2c6cee4b23b3a9e9b5"} Sep 14 11:01:25.597860 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:25.597497 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77b554968f-rrgfr" Sep 14 11:01:25.599169 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:25.599144 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77b554968f-rrgfr" Sep 14 11:01:25.599644 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:25.599623 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" event={"ID":"750c7567-a733-462b-82ba-00c9e6c0065e","Type":"ContainerStarted","Data":"543f8437f7750a743796ad0f68034f4d158009fb46d994d5376f1e43d84be8b6"} Sep 14 11:01:25.599724 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:25.599653 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" event={"ID":"750c7567-a733-462b-82ba-00c9e6c0065e","Type":"ContainerStarted","Data":"c97c742da0b972f8ea02bb02728bb008f45af85c6ba29473cee2a9ba705b239c"} Sep 14 11:01:25.599724 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:25.599667 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" event={"ID":"750c7567-a733-462b-82ba-00c9e6c0065e","Type":"ContainerStarted","Data":"a93c3464f2adb0c205fe65b73080dcb3dcfae3d473d0c618d49cb73bfbcc6b30"} Sep 14 11:01:25.601175 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:25.601156 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nvm9v" event={"ID":"9bd76a15-e9b6-402e-ad38-e1a3ab4a1a18","Type":"ContainerStarted","Data":"8aa9026b63f537b71031ca6541d8e5c781e3ae0c68e89cd74fafcb31fe0658bf"} Sep 14 11:01:25.617021 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:25.616950 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77b554968f-rrgfr" podStartSLOduration=13.973184921 podStartE2EDuration="21.616936588s" totalImagesPullingTime="7.643751667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:25.616297049 +0000 UTC m=+43.679699958" watchObservedRunningTime="2026-09-14 11:01:25.616936588 +0000 UTC m=+43.680339498" Sep 14 11:01:25.617466 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:25.617436 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2md5p" podStartSLOduration=21.262088682 podStartE2EDuration="42.617427009s" totalImagesPullingTime="21.355338327s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:19.61830945 +0000 UTC m=+37.681712363" watchObservedRunningTime="2026-09-14 11:01:25.617427009 +0000 UTC m=+43.680829919" Sep 14 11:01:25.663488 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:25.663444 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" podStartSLOduration=14.034607317 podStartE2EDuration="21.663428139s" totalImagesPullingTime="7.628820822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:25.662530727 +0000 UTC m=+43.725933635" watchObservedRunningTime="2026-09-14 11:01:25.663428139 +0000 UTC m=+43.726831040" Sep 14 11:01:25.682608 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:25.682559 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nvm9v" podStartSLOduration=2.055069061 podStartE2EDuration="9.682545725s" totalImagesPullingTime="7.627476664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:25.681838688 +0000 UTC m=+43.745241600" watchObservedRunningTime="2026-09-14 11:01:25.682545725 +0000 UTC m=+43.745948625" Sep 14 11:01:32.083031 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:32.082967 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:32.083465 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:32.083135 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 11:01:32.083465 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:32.083157 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55555f4b99-6hgh7: secret "image-registry-tls" not found Sep 14 11:01:32.083465 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:32.083212 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls podName:f71e3f64-7c46-4b5f-8e66-8b20d9d32983 nodeName:}" failed. No retries permitted until 2026-09-14 11:01:48.083196344 +0000 UTC m=+66.146599232 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls") pod "image-registry-55555f4b99-6hgh7" (UID: "f71e3f64-7c46-4b5f-8e66-8b20d9d32983") : secret "image-registry-tls" not found Sep 14 11:01:32.183825 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:32.183789 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-metrics-tls\") pod \"dns-default-k264k\" (UID: \"e59abbc8-c9b6-45ec-ae03-a7b534ca02df\") " pod="openshift-dns/dns-default-k264k" Sep 14 11:01:32.183998 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:32.183924 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 11:01:32.183998 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:32.183983 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-metrics-tls podName:e59abbc8-c9b6-45ec-ae03-a7b534ca02df nodeName:}" failed. No retries permitted until 2026-09-14 11:01:48.183963731 +0000 UTC m=+66.247366632 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-metrics-tls") pod "dns-default-k264k" (UID: "e59abbc8-c9b6-45ec-ae03-a7b534ca02df") : secret "dns-default-metrics-tls" not found Sep 14 11:01:36.630954 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:36.630921 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cd76b959d-4hcwn" event={"ID":"54f80caa-892f-4553-88ba-018c70307b68","Type":"ContainerStarted","Data":"5485c5c98e50d8169b64fc9a675f5e76b3b49b8b501661433a5e2b95acb1af31"} Sep 14 11:01:36.653986 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:36.653938 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cd76b959d-4hcwn" podStartSLOduration=13.543296381 podStartE2EDuration="32.653923889s" totalImagesPullingTime="19.110627508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:36.651397337 +0000 UTC m=+54.714800246" watchObservedRunningTime="2026-09-14 11:01:36.653923889 +0000 UTC m=+54.717326826" Sep 14 11:01:44.528764 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:44.528736 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlmvr" Sep 14 11:01:48.097907 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:48.097871 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6547f534-40c9-42c7-b118-37389bd1a650-original-pull-secret\") pod \"global-pull-secret-syncer-96bxc\" (UID: \"6547f534-40c9-42c7-b118-37389bd1a650\") " pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:01:48.098345 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:48.097937 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnvm5\" (UniqueName: \"kubernetes.io/projected/6de42dc8-677e-4919-816e-19952a0ca273-kube-api-access-rnvm5\") pod \"network-check-target-d5x29\" (UID: \"6de42dc8-677e-4919-816e-19952a0ca273\") " pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:01:48.098345 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:48.097977 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:01:48.098345 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:48.098112 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 11:01:48.098345 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:48.098123 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55555f4b99-6hgh7: secret "image-registry-tls" not found Sep 14 11:01:48.098345 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:48.098173 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls podName:f71e3f64-7c46-4b5f-8e66-8b20d9d32983 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:20.098159398 +0000 UTC m=+98.161562286 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls") pod "image-registry-55555f4b99-6hgh7" (UID: "f71e3f64-7c46-4b5f-8e66-8b20d9d32983") : secret "image-registry-tls" not found Sep 14 11:01:48.102739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:48.102722 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 14 11:01:48.102793 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:48.102722 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 14 11:01:48.110209 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:48.110192 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 14 11:01:48.111430 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:48.111409 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6547f534-40c9-42c7-b118-37389bd1a650-original-pull-secret\") pod \"global-pull-secret-syncer-96bxc\" (UID: \"6547f534-40c9-42c7-b118-37389bd1a650\") " pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:01:48.120375 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:48.120355 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnvm5\" (UniqueName: \"kubernetes.io/projected/6de42dc8-677e-4919-816e-19952a0ca273-kube-api-access-rnvm5\") pod \"network-check-target-d5x29\" (UID: \"6de42dc8-677e-4919-816e-19952a0ca273\") " pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:01:48.186638 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:48.186618 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96bxc" Sep 14 11:01:48.198446 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:48.198421 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-metrics-tls\") pod \"dns-default-k264k\" (UID: \"e59abbc8-c9b6-45ec-ae03-a7b534ca02df\") " pod="openshift-dns/dns-default-k264k" Sep 14 11:01:48.198518 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:48.198504 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs\") pod \"network-metrics-daemon-z75dj\" (UID: \"a69c5498-e86f-4c58-940f-1d14fa24a91a\") " pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:01:48.198555 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:48.198525 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 11:01:48.198597 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:48.198578 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-metrics-tls podName:e59abbc8-c9b6-45ec-ae03-a7b534ca02df nodeName:}" failed. No retries permitted until 2026-09-14 11:02:20.198561114 +0000 UTC m=+98.261964023 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-metrics-tls") pod "dns-default-k264k" (UID: "e59abbc8-c9b6-45ec-ae03-a7b534ca02df") : secret "dns-default-metrics-tls" not found Sep 14 11:01:48.198715 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:48.198697 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mckdx\"" Sep 14 11:01:48.205768 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:48.205751 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:01:48.206044 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:48.206030 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 14 11:01:48.209159 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:48.209138 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 14 11:01:48.209260 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:01:48.209182 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs podName:a69c5498-e86f-4c58-940f-1d14fa24a91a nodeName:}" failed. No retries permitted until 2026-09-14 11:02:52.20917015 +0000 UTC m=+130.272573042 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs") pod "network-metrics-daemon-z75dj" (UID: "a69c5498-e86f-4c58-940f-1d14fa24a91a") : secret "metrics-daemon-secret" not found Sep 14 11:01:48.487490 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:48.487452 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-96bxc"] Sep 14 11:01:48.490004 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:01:48.489976 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6547f534_40c9_42c7_b118_37389bd1a650.slice/crio-d724dce6f3260df43513cc554053435e0bcb5ab4b65cec2721a90210291e25c3 WatchSource:0}: Error finding container d724dce6f3260df43513cc554053435e0bcb5ab4b65cec2721a90210291e25c3: Status 404 returned error can't find the container with id d724dce6f3260df43513cc554053435e0bcb5ab4b65cec2721a90210291e25c3 Sep 14 11:01:48.507534 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:48.507510 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d5x29"] Sep 14 11:01:48.509644 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:01:48.509624 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6de42dc8_677e_4919_816e_19952a0ca273.slice/crio-3a7ede68c8dcb293396f9fb0b4db71ddbf5fdaa9a352735406ed53010c0c0bad WatchSource:0}: Error finding container 3a7ede68c8dcb293396f9fb0b4db71ddbf5fdaa9a352735406ed53010c0c0bad: Status 404 returned error can't find the container with id 3a7ede68c8dcb293396f9fb0b4db71ddbf5fdaa9a352735406ed53010c0c0bad Sep 14 11:01:48.665432 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:48.665406 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d5x29" event={"ID":"6de42dc8-677e-4919-816e-19952a0ca273","Type":"ContainerStarted","Data":"3a7ede68c8dcb293396f9fb0b4db71ddbf5fdaa9a352735406ed53010c0c0bad"} Sep 14 11:01:48.666352 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:48.666333 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-96bxc" event={"ID":"6547f534-40c9-42c7-b118-37389bd1a650","Type":"ContainerStarted","Data":"d724dce6f3260df43513cc554053435e0bcb5ab4b65cec2721a90210291e25c3"} Sep 14 11:01:53.685961 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:53.685917 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d5x29" event={"ID":"6de42dc8-677e-4919-816e-19952a0ca273","Type":"ContainerStarted","Data":"2e7b2075739522a9a89d1f203eaa3e42244825cb0ea60194c724a1b8f1a1421f"} Sep 14 11:01:53.686375 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:53.686005 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:01:53.687600 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:53.687572 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-96bxc" event={"ID":"6547f534-40c9-42c7-b118-37389bd1a650","Type":"ContainerStarted","Data":"e3d690497fb1b78112b70d3289b189ff13c58968210dec4a98f54f522b4e81b4"} Sep 14 11:01:53.708082 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:53.708041 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-d5x29" podStartSLOduration=66.170657007 podStartE2EDuration="1m10.708028165s" totalImagesPullingTime="4.537371158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:53.707274984 +0000 UTC m=+71.770677895" watchObservedRunningTime="2026-09-14 11:01:53.708028165 +0000 UTC m=+71.771431076" Sep 14 11:01:53.727278 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:01:53.727239 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-96bxc" podStartSLOduration=66.170008594 podStartE2EDuration="1m10.727227217s" totalImagesPullingTime="4.557218623s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:53.726219061 +0000 UTC m=+71.789621971" watchObservedRunningTime="2026-09-14 11:01:53.727227217 +0000 UTC m=+71.790630126" Sep 14 11:02:06.122272 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:06.122162 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-24mtb"] Sep 14 11:02:06.125794 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:06.125746 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:06.128327 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:06.128307 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 14 11:02:06.129159 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:06.129143 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nzc8q\"" Sep 14 11:02:06.129242 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:06.129176 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 14 11:02:06.129242 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:06.129185 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 14 11:02:06.129242 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:06.129194 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 14 11:02:06.144503 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:06.144479 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-24mtb"] Sep 14 11:02:06.221468 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:06.221442 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24mtb\" (UID: \"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4\") " pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:06.221468 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:06.221468 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-crio-socket\") pod \"insights-runtime-extractor-24mtb\" (UID: \"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4\") " pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:06.221628 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:06.221486 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24mtb\" (UID: \"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4\") " pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:06.221628 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:06.221509 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-data-volume\") pod \"insights-runtime-extractor-24mtb\" (UID: \"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4\") " pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:06.221628 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:06.221608 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm5vk\" (UniqueName: \"kubernetes.io/projected/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-kube-api-access-xm5vk\") pod \"insights-runtime-extractor-24mtb\" (UID: \"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4\") " pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:06.322045 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:06.322015 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm5vk\" (UniqueName: \"kubernetes.io/projected/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-kube-api-access-xm5vk\") pod \"insights-runtime-extractor-24mtb\" (UID: \"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4\") " pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:06.322164 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:06.322059 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24mtb\" (UID: \"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4\") " pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:06.322164 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:06.322077 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-crio-socket\") pod \"insights-runtime-extractor-24mtb\" (UID: \"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4\") " pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:06.322164 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:06.322115 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24mtb\" (UID: \"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4\") " pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:06.322164 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:06.322150 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-data-volume\") pod \"insights-runtime-extractor-24mtb\" (UID: \"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4\") " pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:06.322354 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:02:06.322165 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 11:02:06.322354 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:06.322178 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-crio-socket\") pod \"insights-runtime-extractor-24mtb\" (UID: \"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4\") " pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:06.322354 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:02:06.322243 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-kube-rbac-proxy-cm podName:b49b5ae9-93d6-4fa5-afa0-3c6b64766be4 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:06.822225896 +0000 UTC m=+84.885628799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-24mtb" (UID: "b49b5ae9-93d6-4fa5-afa0-3c6b64766be4") : configmap "kube-rbac-proxy" not found Sep 14 11:02:06.322354 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:02:06.322295 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 11:02:06.322498 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:02:06.322369 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-insights-runtime-extractor-tls podName:b49b5ae9-93d6-4fa5-afa0-3c6b64766be4 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:06.822350948 +0000 UTC m=+84.885753837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24mtb" (UID: "b49b5ae9-93d6-4fa5-afa0-3c6b64766be4") : secret "insights-runtime-extractor-tls" not found Sep 14 11:02:06.322498 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:06.322478 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-data-volume\") pod \"insights-runtime-extractor-24mtb\" (UID: \"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4\") " pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:06.333845 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:06.333818 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm5vk\" (UniqueName: \"kubernetes.io/projected/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-kube-api-access-xm5vk\") pod \"insights-runtime-extractor-24mtb\" (UID: \"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4\") " pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:06.825945 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:06.825918 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24mtb\" (UID: \"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4\") " pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:06.826115 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:06.825953 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24mtb\" (UID: \"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4\") " pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:06.826115 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:02:06.826035 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 11:02:06.826115 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:02:06.826071 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 11:02:06.826211 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:02:06.826128 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-kube-rbac-proxy-cm podName:b49b5ae9-93d6-4fa5-afa0-3c6b64766be4 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:07.826108587 +0000 UTC m=+85.889511492 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-24mtb" (UID: "b49b5ae9-93d6-4fa5-afa0-3c6b64766be4") : configmap "kube-rbac-proxy" not found Sep 14 11:02:06.826211 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:02:06.826143 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-insights-runtime-extractor-tls podName:b49b5ae9-93d6-4fa5-afa0-3c6b64766be4 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:07.826137402 +0000 UTC m=+85.889540294 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24mtb" (UID: "b49b5ae9-93d6-4fa5-afa0-3c6b64766be4") : secret "insights-runtime-extractor-tls" not found Sep 14 11:02:07.833703 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:07.833658 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24mtb\" (UID: \"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4\") " pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:07.833703 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:07.833706 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24mtb\" (UID: \"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4\") " pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:07.834186 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:02:07.833778 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 11:02:07.834186 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:02:07.833837 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-kube-rbac-proxy-cm podName:b49b5ae9-93d6-4fa5-afa0-3c6b64766be4 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:09.833823507 +0000 UTC m=+87.897226395 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-24mtb" (UID: "b49b5ae9-93d6-4fa5-afa0-3c6b64766be4") : configmap "kube-rbac-proxy" not found Sep 14 11:02:07.834186 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:02:07.833844 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 11:02:07.834186 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:02:07.833895 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-insights-runtime-extractor-tls podName:b49b5ae9-93d6-4fa5-afa0-3c6b64766be4 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:09.833880387 +0000 UTC m=+87.897283280 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24mtb" (UID: "b49b5ae9-93d6-4fa5-afa0-3c6b64766be4") : secret "insights-runtime-extractor-tls" not found Sep 14 11:02:08.351732 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:08.351707 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mcxhz_2f816499-fb81-4f59-bb3f-986f34d9c571/dns-node-resolver/0.log" Sep 14 11:02:08.949377 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:08.949349 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5qhj6_27fdd2fa-1378-4960-bd86-fd1dd84516c3/node-ca/0.log" Sep 14 11:02:09.847945 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:09.847915 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24mtb\" (UID: \"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4\") " pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:09.847945 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:09.847946 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24mtb\" (UID: \"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4\") " pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:09.848177 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:02:09.848035 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 11:02:09.848177 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:02:09.848163 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 11:02:09.848248 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:02:09.848165 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-kube-rbac-proxy-cm podName:b49b5ae9-93d6-4fa5-afa0-3c6b64766be4 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:13.848145609 +0000 UTC m=+91.911548517 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-24mtb" (UID: "b49b5ae9-93d6-4fa5-afa0-3c6b64766be4") : configmap "kube-rbac-proxy" not found Sep 14 11:02:09.848248 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:02:09.848212 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-insights-runtime-extractor-tls podName:b49b5ae9-93d6-4fa5-afa0-3c6b64766be4 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:13.848201765 +0000 UTC m=+91.911604658 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24mtb" (UID: "b49b5ae9-93d6-4fa5-afa0-3c6b64766be4") : secret "insights-runtime-extractor-tls" not found Sep 14 11:02:10.146644 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:10.146619 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nvm9v_9bd76a15-e9b6-402e-ad38-e1a3ab4a1a18/serve-healthcheck-canary/0.log" Sep 14 11:02:13.874611 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:13.874580 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24mtb\" (UID: \"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4\") " pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:13.874611 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:13.874617 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24mtb\" (UID: \"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4\") " pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:13.875064 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:02:13.874701 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 11:02:13.875064 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:02:13.874745 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 11:02:13.875064 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:02:13.874768 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-kube-rbac-proxy-cm podName:b49b5ae9-93d6-4fa5-afa0-3c6b64766be4 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:21.874751712 +0000 UTC m=+99.938154620 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-24mtb" (UID: "b49b5ae9-93d6-4fa5-afa0-3c6b64766be4") : configmap "kube-rbac-proxy" not found Sep 14 11:02:13.875064 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:02:13.874785 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-insights-runtime-extractor-tls podName:b49b5ae9-93d6-4fa5-afa0-3c6b64766be4 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:21.874774592 +0000 UTC m=+99.938177480 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24mtb" (UID: "b49b5ae9-93d6-4fa5-afa0-3c6b64766be4") : secret "insights-runtime-extractor-tls" not found Sep 14 11:02:20.116725 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:20.116694 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:02:20.118943 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:20.118920 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls\") pod \"image-registry-55555f4b99-6hgh7\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:02:20.186228 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:20.186208 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nd2nr\"" Sep 14 11:02:20.194524 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:20.194505 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:02:20.217196 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:20.217173 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-metrics-tls\") pod \"dns-default-k264k\" (UID: \"e59abbc8-c9b6-45ec-ae03-a7b534ca02df\") " pod="openshift-dns/dns-default-k264k" Sep 14 11:02:20.219226 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:20.219207 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e59abbc8-c9b6-45ec-ae03-a7b534ca02df-metrics-tls\") pod \"dns-default-k264k\" (UID: \"e59abbc8-c9b6-45ec-ae03-a7b534ca02df\") " pod="openshift-dns/dns-default-k264k" Sep 14 11:02:20.261843 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:20.261824 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zpbn2\"" Sep 14 11:02:20.270041 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:20.270023 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k264k" Sep 14 11:02:20.467260 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:20.467238 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55555f4b99-6hgh7"] Sep 14 11:02:20.469259 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:02:20.469223 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71e3f64_7c46_4b5f_8e66_8b20d9d32983.slice/crio-dd4b2c765082a06919238f852bd03cf3158ec97f8c05eea283a529915559a89d WatchSource:0}: Error finding container dd4b2c765082a06919238f852bd03cf3158ec97f8c05eea283a529915559a89d: Status 404 returned error can't find the container with id dd4b2c765082a06919238f852bd03cf3158ec97f8c05eea283a529915559a89d Sep 14 11:02:20.540650 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:20.540627 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k264k"] Sep 14 11:02:20.545009 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:02:20.544986 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59abbc8_c9b6_45ec_ae03_a7b534ca02df.slice/crio-89b5623992584966334a56b4a91feb957b07af83daf2ade62f51b1645e4d2651 WatchSource:0}: Error finding container 89b5623992584966334a56b4a91feb957b07af83daf2ade62f51b1645e4d2651: Status 404 returned error can't find the container with id 89b5623992584966334a56b4a91feb957b07af83daf2ade62f51b1645e4d2651 Sep 14 11:02:20.767955 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:20.767876 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k264k" event={"ID":"e59abbc8-c9b6-45ec-ae03-a7b534ca02df","Type":"ContainerStarted","Data":"89b5623992584966334a56b4a91feb957b07af83daf2ade62f51b1645e4d2651"} Sep 14 11:02:20.769523 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:20.769495 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" event={"ID":"f71e3f64-7c46-4b5f-8e66-8b20d9d32983","Type":"ContainerStarted","Data":"eb9b79655d9e2f54e7e325f6d1c2afdd5a6dc3338aaf229c9f7cf2570229ebe2"} Sep 14 11:02:20.769660 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:20.769540 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" event={"ID":"f71e3f64-7c46-4b5f-8e66-8b20d9d32983","Type":"ContainerStarted","Data":"dd4b2c765082a06919238f852bd03cf3158ec97f8c05eea283a529915559a89d"} Sep 14 11:02:20.769660 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:20.769612 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:02:20.791722 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:20.791678 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" podStartSLOduration=97.791661983 podStartE2EDuration="1m37.791661983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:20.791078009 +0000 UTC m=+98.854480916" watchObservedRunningTime="2026-09-14 11:02:20.791661983 +0000 UTC m=+98.855064920" Sep 14 11:02:21.931488 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:21.931458 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24mtb\" (UID: \"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4\") " pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:21.931488 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:21.931491 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24mtb\" (UID: \"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4\") " pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:21.932015 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:21.931984 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24mtb\" (UID: \"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4\") " pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:21.933928 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:21.933906 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b49b5ae9-93d6-4fa5-afa0-3c6b64766be4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24mtb\" (UID: \"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4\") " pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:22.034035 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:22.034012 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-24mtb" Sep 14 11:02:22.340140 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:22.340056 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-24mtb"] Sep 14 11:02:22.342721 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:02:22.342691 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb49b5ae9_93d6_4fa5_afa0_3c6b64766be4.slice/crio-33f5a5780770090e3ee2c29b6be24a2fed33e7e4fe2b409a2e8a0e86e3211841 WatchSource:0}: Error finding container 33f5a5780770090e3ee2c29b6be24a2fed33e7e4fe2b409a2e8a0e86e3211841: Status 404 returned error can't find the container with id 33f5a5780770090e3ee2c29b6be24a2fed33e7e4fe2b409a2e8a0e86e3211841 Sep 14 11:02:22.778871 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:22.778827 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-24mtb" event={"ID":"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4","Type":"ContainerStarted","Data":"e3c4efe690822bc82185dbbe4c43028ad0894f867570b79e3c3bceed2d200de3"} Sep 14 11:02:22.779068 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:22.778901 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-24mtb" event={"ID":"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4","Type":"ContainerStarted","Data":"33f5a5780770090e3ee2c29b6be24a2fed33e7e4fe2b409a2e8a0e86e3211841"} Sep 14 11:02:24.692905 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:24.692877 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-d5x29" Sep 14 11:02:24.786374 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:24.786348 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k264k" event={"ID":"e59abbc8-c9b6-45ec-ae03-a7b534ca02df","Type":"ContainerStarted","Data":"66df83a381ae1a1ed89786b1072a05e03b27cbad4381aadbfb37523eca1545e2"} Sep 14 11:02:24.786514 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:24.786390 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k264k" event={"ID":"e59abbc8-c9b6-45ec-ae03-a7b534ca02df","Type":"ContainerStarted","Data":"eadae361db1652248fc013d3db78d810f3c270a53b330d166328cb9425b5919f"} Sep 14 11:02:24.786514 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:24.786457 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-k264k" Sep 14 11:02:24.788078 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:24.788057 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-24mtb" event={"ID":"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4","Type":"ContainerStarted","Data":"4df488502881267adff2cd4c620f34ae50b3416a729b2658454fe94f03798697"} Sep 14 11:02:24.806647 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:24.806601 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-k264k" podStartSLOduration=65.2599732 podStartE2EDuration="1m8.806584303s" totalImagesPullingTime="3.546611103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:24.804804092 +0000 UTC m=+102.868207003" watchObservedRunningTime="2026-09-14 11:02:24.806584303 +0000 UTC m=+102.869987213" Sep 14 11:02:27.801479 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:27.801446 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-24mtb" event={"ID":"b49b5ae9-93d6-4fa5-afa0-3c6b64766be4","Type":"ContainerStarted","Data":"c659092de8e72182e899f87de955e7952fca3693ea9013ed462efe72193ef956"} Sep 14 11:02:27.833862 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:27.833814 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-24mtb" podStartSLOduration=17.594930104 podStartE2EDuration="21.833798303s" totalImagesPullingTime="4.238868199s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:02:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:27.83302738 +0000 UTC m=+105.896430290" watchObservedRunningTime="2026-09-14 11:02:27.833798303 +0000 UTC m=+105.897201213" Sep 14 11:02:34.793676 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:34.793647 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-k264k" Sep 14 11:02:40.199749 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:40.199719 2556 patch_prober.go:28] interesting pod/image-registry-55555f4b99-6hgh7 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 14 11:02:40.200132 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:40.199764 2556 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" podUID="f71e3f64-7c46-4b5f-8e66-8b20d9d32983" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 14 11:02:41.778328 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:41.778303 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:02:44.331020 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.330988 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2kcvz"] Sep 14 11:02:44.333327 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.333310 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.337041 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.337023 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 14 11:02:44.337355 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.337334 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 14 11:02:44.337435 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.337340 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-56dhl\"" Sep 14 11:02:44.337477 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.337339 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 14 11:02:44.338430 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.338414 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 14 11:02:44.338522 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.338415 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 14 11:02:44.338522 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.338416 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 14 11:02:44.479025 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.478996 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/748fb694-37f5-47cb-9fad-4cc1e1d7421f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.479260 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.479034 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/748fb694-37f5-47cb-9fad-4cc1e1d7421f-root\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.479260 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.479066 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/748fb694-37f5-47cb-9fad-4cc1e1d7421f-metrics-client-ca\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.479260 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.479146 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/748fb694-37f5-47cb-9fad-4cc1e1d7421f-node-exporter-textfile\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.479260 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.479186 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2nvb\" (UniqueName: \"kubernetes.io/projected/748fb694-37f5-47cb-9fad-4cc1e1d7421f-kube-api-access-d2nvb\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.479260 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.479211 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/748fb694-37f5-47cb-9fad-4cc1e1d7421f-node-exporter-wtmp\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.479260 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.479226 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/748fb694-37f5-47cb-9fad-4cc1e1d7421f-node-exporter-tls\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.479260 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.479242 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/748fb694-37f5-47cb-9fad-4cc1e1d7421f-node-exporter-accelerators-collector-config\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.479479 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.479282 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/748fb694-37f5-47cb-9fad-4cc1e1d7421f-sys\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.580432 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.580405 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/748fb694-37f5-47cb-9fad-4cc1e1d7421f-metrics-client-ca\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.580601 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.580445 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/748fb694-37f5-47cb-9fad-4cc1e1d7421f-node-exporter-textfile\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.580601 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.580463 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2nvb\" (UniqueName: \"kubernetes.io/projected/748fb694-37f5-47cb-9fad-4cc1e1d7421f-kube-api-access-d2nvb\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.580722 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.580591 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/748fb694-37f5-47cb-9fad-4cc1e1d7421f-node-exporter-wtmp\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.580722 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.580637 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/748fb694-37f5-47cb-9fad-4cc1e1d7421f-node-exporter-tls\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.580722 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.580667 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/748fb694-37f5-47cb-9fad-4cc1e1d7421f-node-exporter-accelerators-collector-config\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.580722 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.580707 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/748fb694-37f5-47cb-9fad-4cc1e1d7421f-sys\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.580924 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:02:44.580759 2556 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 14 11:02:44.580924 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.580773 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/748fb694-37f5-47cb-9fad-4cc1e1d7421f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.580924 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.580799 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/748fb694-37f5-47cb-9fad-4cc1e1d7421f-sys\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.580924 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.580810 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/748fb694-37f5-47cb-9fad-4cc1e1d7421f-root\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.580924 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.580765 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/748fb694-37f5-47cb-9fad-4cc1e1d7421f-node-exporter-wtmp\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.580924 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.580817 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/748fb694-37f5-47cb-9fad-4cc1e1d7421f-node-exporter-textfile\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.580924 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:02:44.580843 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/748fb694-37f5-47cb-9fad-4cc1e1d7421f-node-exporter-tls podName:748fb694-37f5-47cb-9fad-4cc1e1d7421f nodeName:}" failed. No retries permitted until 2026-09-14 11:02:45.080822948 +0000 UTC m=+123.144225855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/748fb694-37f5-47cb-9fad-4cc1e1d7421f-node-exporter-tls") pod "node-exporter-2kcvz" (UID: "748fb694-37f5-47cb-9fad-4cc1e1d7421f") : secret "node-exporter-tls" not found Sep 14 11:02:44.580924 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.580864 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/748fb694-37f5-47cb-9fad-4cc1e1d7421f-root\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.581342 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.581018 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/748fb694-37f5-47cb-9fad-4cc1e1d7421f-metrics-client-ca\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.581342 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.581282 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/748fb694-37f5-47cb-9fad-4cc1e1d7421f-node-exporter-accelerators-collector-config\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.582950 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.582931 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/748fb694-37f5-47cb-9fad-4cc1e1d7421f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:44.590555 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:44.590536 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2nvb\" (UniqueName: \"kubernetes.io/projected/748fb694-37f5-47cb-9fad-4cc1e1d7421f-kube-api-access-d2nvb\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:45.084657 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:45.084624 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/748fb694-37f5-47cb-9fad-4cc1e1d7421f-node-exporter-tls\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:45.086962 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:45.086938 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/748fb694-37f5-47cb-9fad-4cc1e1d7421f-node-exporter-tls\") pod \"node-exporter-2kcvz\" (UID: \"748fb694-37f5-47cb-9fad-4cc1e1d7421f\") " pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:45.241671 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:45.241643 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2kcvz" Sep 14 11:02:45.252587 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:02:45.252559 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod748fb694_37f5_47cb_9fad_4cc1e1d7421f.slice/crio-9e601fd7673c2d3b7b892c3743a2f832ca03e3fee83b9528ffec9ebb9a2f8c44 WatchSource:0}: Error finding container 9e601fd7673c2d3b7b892c3743a2f832ca03e3fee83b9528ffec9ebb9a2f8c44: Status 404 returned error can't find the container with id 9e601fd7673c2d3b7b892c3743a2f832ca03e3fee83b9528ffec9ebb9a2f8c44 Sep 14 11:02:45.855282 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:45.855248 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2kcvz" event={"ID":"748fb694-37f5-47cb-9fad-4cc1e1d7421f","Type":"ContainerStarted","Data":"9e601fd7673c2d3b7b892c3743a2f832ca03e3fee83b9528ffec9ebb9a2f8c44"} Sep 14 11:02:46.860235 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:46.860201 2556 generic.go:356] "Generic (PLEG): container finished" podID="748fb694-37f5-47cb-9fad-4cc1e1d7421f" containerID="396e8f37efee6dcb84e0fcd996510b9bd55081c57c56cbdad1331f5e1a6e0cf4" exitCode=0 Sep 14 11:02:46.860698 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:46.860254 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2kcvz" event={"ID":"748fb694-37f5-47cb-9fad-4cc1e1d7421f","Type":"ContainerDied","Data":"396e8f37efee6dcb84e0fcd996510b9bd55081c57c56cbdad1331f5e1a6e0cf4"} Sep 14 11:02:47.865634 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:47.865598 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2kcvz" event={"ID":"748fb694-37f5-47cb-9fad-4cc1e1d7421f","Type":"ContainerStarted","Data":"c50d648616d56b1bf37b5d17b8c5e7e207cb3d5a16a34fe69f9259e7790a194b"} Sep 14 11:02:47.866145 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:47.865653 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2kcvz" event={"ID":"748fb694-37f5-47cb-9fad-4cc1e1d7421f","Type":"ContainerStarted","Data":"a9d21d7ee906961b83180bf3e9d7291e58b2d1136822c2c66cc82dd43c475e25"} Sep 14 11:02:47.889235 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:47.889193 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2kcvz" podStartSLOduration=2.91212831 podStartE2EDuration="3.88917689s" totalImagesPullingTime="915.432752ms" totalInitContainerRuntime="61.615828ms" isStatefulPod=true podCreationTimestamp="2026-09-14 11:02:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:47.887297987 +0000 UTC m=+125.950700898" watchObservedRunningTime="2026-09-14 11:02:47.88917689 +0000 UTC m=+125.952579801" Sep 14 11:02:49.353708 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:49.353669 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55555f4b99-6hgh7"] Sep 14 11:02:52.241717 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:52.241669 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs\") pod \"network-metrics-daemon-z75dj\" (UID: \"a69c5498-e86f-4c58-940f-1d14fa24a91a\") " pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:02:52.243874 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:52.243854 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a69c5498-e86f-4c58-940f-1d14fa24a91a-metrics-certs\") pod \"network-metrics-daemon-z75dj\" (UID: \"a69c5498-e86f-4c58-940f-1d14fa24a91a\") " pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:02:52.405266 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:52.405237 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-npv4p\"" Sep 14 11:02:52.409917 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:52.409903 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z75dj" Sep 14 11:02:52.688656 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:02:52.688623 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda69c5498_e86f_4c58_940f_1d14fa24a91a.slice/crio-6444ab6fafaddd1b8bdfe60e4b50dfbd33e60d61afc000e121b455b015ef2048 WatchSource:0}: Error finding container 6444ab6fafaddd1b8bdfe60e4b50dfbd33e60d61afc000e121b455b015ef2048: Status 404 returned error can't find the container with id 6444ab6fafaddd1b8bdfe60e4b50dfbd33e60d61afc000e121b455b015ef2048 Sep 14 11:02:52.689291 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:52.689262 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z75dj"] Sep 14 11:02:52.881925 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:52.881892 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z75dj" event={"ID":"a69c5498-e86f-4c58-940f-1d14fa24a91a","Type":"ContainerStarted","Data":"6444ab6fafaddd1b8bdfe60e4b50dfbd33e60d61afc000e121b455b015ef2048"} Sep 14 11:02:53.887460 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:53.887424 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z75dj" event={"ID":"a69c5498-e86f-4c58-940f-1d14fa24a91a","Type":"ContainerStarted","Data":"27f6e341f5a02d9fb6d29257c263e54d4cce065fdb4fe248cc3f25cb4077501a"} Sep 14 11:02:53.887868 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:53.887476 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z75dj" event={"ID":"a69c5498-e86f-4c58-940f-1d14fa24a91a","Type":"ContainerStarted","Data":"af313d8ff567f8139b74cd03c3b8004a86eb7e42674477e0e9b52aec41dcfa32"} Sep 14 11:02:53.917162 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:53.917111 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-z75dj" podStartSLOduration=130.146400518 podStartE2EDuration="2m10.917074973s" totalImagesPullingTime="770.674455ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:53.915904198 +0000 UTC m=+131.979307109" watchObservedRunningTime="2026-09-14 11:02:53.917074973 +0000 UTC m=+131.980477885" Sep 14 11:02:56.633110 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:02:56.633047 2556 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" podUID="750c7567-a733-462b-82ba-00c9e6c0065e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 14 11:03:06.632675 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:06.632637 2556 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" podUID="750c7567-a733-462b-82ba-00c9e6c0065e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 14 11:03:14.373574 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.373511 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" podUID="f71e3f64-7c46-4b5f-8e66-8b20d9d32983" containerName="registry" containerID="cri-o://eb9b79655d9e2f54e7e325f6d1c2afdd5a6dc3338aaf229c9f7cf2570229ebe2" gracePeriod=30 Sep 14 11:03:14.612035 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.612010 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:03:14.691626 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.691547 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kk9s8\" (UniqueName: \"kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-kube-api-access-kk9s8\") pod \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " Sep 14 11:03:14.691626 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.691583 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-ca-trust-extracted\") pod \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " Sep 14 11:03:14.691626 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.691606 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls\") pod \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " Sep 14 11:03:14.691882 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.691640 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-bound-sa-token\") pod \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " Sep 14 11:03:14.691882 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.691670 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-certificates\") pod \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " Sep 14 11:03:14.691882 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.691702 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-trusted-ca\") pod \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " Sep 14 11:03:14.691882 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.691729 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-image-registry-private-configuration\") pod \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " Sep 14 11:03:14.691882 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.691784 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-installation-pull-secrets\") pod \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\" (UID: \"f71e3f64-7c46-4b5f-8e66-8b20d9d32983\") " Sep 14 11:03:14.692322 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.692295 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f71e3f64-7c46-4b5f-8e66-8b20d9d32983" (UID: "f71e3f64-7c46-4b5f-8e66-8b20d9d32983"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:03:14.692399 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.692324 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f71e3f64-7c46-4b5f-8e66-8b20d9d32983" (UID: "f71e3f64-7c46-4b5f-8e66-8b20d9d32983"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:03:14.694374 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.694258 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f71e3f64-7c46-4b5f-8e66-8b20d9d32983" (UID: "f71e3f64-7c46-4b5f-8e66-8b20d9d32983"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:03:14.694501 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.694474 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-kube-api-access-kk9s8" (OuterVolumeSpecName: "kube-api-access-kk9s8") pod "f71e3f64-7c46-4b5f-8e66-8b20d9d32983" (UID: "f71e3f64-7c46-4b5f-8e66-8b20d9d32983"). InnerVolumeSpecName "kube-api-access-kk9s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:03:14.694572 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.694550 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f71e3f64-7c46-4b5f-8e66-8b20d9d32983" (UID: "f71e3f64-7c46-4b5f-8e66-8b20d9d32983"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:03:14.694654 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.694626 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f71e3f64-7c46-4b5f-8e66-8b20d9d32983" (UID: "f71e3f64-7c46-4b5f-8e66-8b20d9d32983"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:03:14.694957 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.694932 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f71e3f64-7c46-4b5f-8e66-8b20d9d32983" (UID: "f71e3f64-7c46-4b5f-8e66-8b20d9d32983"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:03:14.700192 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.700166 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f71e3f64-7c46-4b5f-8e66-8b20d9d32983" (UID: "f71e3f64-7c46-4b5f-8e66-8b20d9d32983"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:03:14.793135 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.793111 2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-trusted-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:03:14.793135 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.793135 2556 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-image-registry-private-configuration\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:03:14.793278 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.793145 2556 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-installation-pull-secrets\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:03:14.793278 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.793155 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk9s8\" (UniqueName: \"kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-kube-api-access-kk9s8\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:03:14.793278 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.793164 2556 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-ca-trust-extracted\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:03:14.793278 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.793172 2556 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:03:14.793278 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.793180 2556 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-bound-sa-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:03:14.793278 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.793188 2556 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f71e3f64-7c46-4b5f-8e66-8b20d9d32983-registry-certificates\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:03:14.961199 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.961107 2556 generic.go:356] "Generic (PLEG): container finished" podID="f71e3f64-7c46-4b5f-8e66-8b20d9d32983" containerID="eb9b79655d9e2f54e7e325f6d1c2afdd5a6dc3338aaf229c9f7cf2570229ebe2" exitCode=0 Sep 14 11:03:14.961199 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.961150 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" Sep 14 11:03:14.961199 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.961182 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" event={"ID":"f71e3f64-7c46-4b5f-8e66-8b20d9d32983","Type":"ContainerDied","Data":"eb9b79655d9e2f54e7e325f6d1c2afdd5a6dc3338aaf229c9f7cf2570229ebe2"} Sep 14 11:03:14.961473 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.961222 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55555f4b99-6hgh7" event={"ID":"f71e3f64-7c46-4b5f-8e66-8b20d9d32983","Type":"ContainerDied","Data":"dd4b2c765082a06919238f852bd03cf3158ec97f8c05eea283a529915559a89d"} Sep 14 11:03:14.961473 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.961242 2556 scope.go:117] "RemoveContainer" containerID="eb9b79655d9e2f54e7e325f6d1c2afdd5a6dc3338aaf229c9f7cf2570229ebe2" Sep 14 11:03:14.969436 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.969415 2556 scope.go:117] "RemoveContainer" containerID="eb9b79655d9e2f54e7e325f6d1c2afdd5a6dc3338aaf229c9f7cf2570229ebe2" Sep 14 11:03:14.969680 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:03:14.969662 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb9b79655d9e2f54e7e325f6d1c2afdd5a6dc3338aaf229c9f7cf2570229ebe2\": container with ID starting with eb9b79655d9e2f54e7e325f6d1c2afdd5a6dc3338aaf229c9f7cf2570229ebe2 not found: ID does not exist" containerID="eb9b79655d9e2f54e7e325f6d1c2afdd5a6dc3338aaf229c9f7cf2570229ebe2" Sep 14 11:03:14.969725 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.969687 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb9b79655d9e2f54e7e325f6d1c2afdd5a6dc3338aaf229c9f7cf2570229ebe2"} err="failed to get container status \"eb9b79655d9e2f54e7e325f6d1c2afdd5a6dc3338aaf229c9f7cf2570229ebe2\": rpc error: code = NotFound desc = could not find container \"eb9b79655d9e2f54e7e325f6d1c2afdd5a6dc3338aaf229c9f7cf2570229ebe2\": container with ID starting with eb9b79655d9e2f54e7e325f6d1c2afdd5a6dc3338aaf229c9f7cf2570229ebe2 not found: ID does not exist" Sep 14 11:03:14.988100 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.988056 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55555f4b99-6hgh7"] Sep 14 11:03:14.991016 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:14.990993 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-55555f4b99-6hgh7"] Sep 14 11:03:16.381072 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:16.381038 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f71e3f64-7c46-4b5f-8e66-8b20d9d32983" path="/var/lib/kubelet/pods/f71e3f64-7c46-4b5f-8e66-8b20d9d32983/volumes" Sep 14 11:03:16.632211 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:16.632127 2556 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" podUID="750c7567-a733-462b-82ba-00c9e6c0065e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 14 11:03:16.632211 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:16.632198 2556 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" Sep 14 11:03:16.632710 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:16.632686 2556 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"543f8437f7750a743796ad0f68034f4d158009fb46d994d5376f1e43d84be8b6"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" containerMessage="Container service-proxy failed liveness probe, will be restarted" Sep 14 11:03:16.632789 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:16.632729 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" podUID="750c7567-a733-462b-82ba-00c9e6c0065e" containerName="service-proxy" containerID="cri-o://543f8437f7750a743796ad0f68034f4d158009fb46d994d5376f1e43d84be8b6" gracePeriod=30 Sep 14 11:03:16.971554 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:16.971468 2556 generic.go:356] "Generic (PLEG): container finished" podID="750c7567-a733-462b-82ba-00c9e6c0065e" containerID="543f8437f7750a743796ad0f68034f4d158009fb46d994d5376f1e43d84be8b6" exitCode=2 Sep 14 11:03:16.971554 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:16.971531 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" event={"ID":"750c7567-a733-462b-82ba-00c9e6c0065e","Type":"ContainerDied","Data":"543f8437f7750a743796ad0f68034f4d158009fb46d994d5376f1e43d84be8b6"} Sep 14 11:03:16.971713 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:03:16.971585 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df45887fb-r4dw9" event={"ID":"750c7567-a733-462b-82ba-00c9e6c0065e","Type":"ContainerStarted","Data":"5e93cf9f4fafca2a09d7a982fc94bcc9690a3ea0c608ff401b28a03136d68163"} Sep 14 11:05:35.197157 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:05:35.197125 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r7s7d"] Sep 14 11:05:35.197581 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:05:35.197339 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f71e3f64-7c46-4b5f-8e66-8b20d9d32983" containerName="registry" Sep 14 11:05:35.197581 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:05:35.197349 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f71e3f64-7c46-4b5f-8e66-8b20d9d32983" containerName="registry" Sep 14 11:05:35.197581 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:05:35.197388 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f71e3f64-7c46-4b5f-8e66-8b20d9d32983" containerName="registry" Sep 14 11:05:35.199051 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:05:35.199033 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r7s7d" Sep 14 11:05:35.201515 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:05:35.201484 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 14 11:05:35.201639 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:05:35.201514 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 14 11:05:35.202482 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:05:35.202466 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-scgsn\"" Sep 14 11:05:35.215831 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:05:35.215811 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r7s7d"] Sep 14 11:05:35.291438 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:05:35.291411 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76cqj\" (UniqueName: \"kubernetes.io/projected/3002f8b0-31d8-4b91-a90e-c5d7ea43cfdb-kube-api-access-76cqj\") pod \"openshift-lws-operator-fd8ccff4c-r7s7d\" (UID: \"3002f8b0-31d8-4b91-a90e-c5d7ea43cfdb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r7s7d" Sep 14 11:05:35.291579 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:05:35.291451 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3002f8b0-31d8-4b91-a90e-c5d7ea43cfdb-tmp\") pod \"openshift-lws-operator-fd8ccff4c-r7s7d\" (UID: \"3002f8b0-31d8-4b91-a90e-c5d7ea43cfdb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r7s7d" Sep 14 11:05:35.392193 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:05:35.392166 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76cqj\" (UniqueName: \"kubernetes.io/projected/3002f8b0-31d8-4b91-a90e-c5d7ea43cfdb-kube-api-access-76cqj\") pod \"openshift-lws-operator-fd8ccff4c-r7s7d\" (UID: \"3002f8b0-31d8-4b91-a90e-c5d7ea43cfdb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r7s7d" Sep 14 11:05:35.392307 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:05:35.392209 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3002f8b0-31d8-4b91-a90e-c5d7ea43cfdb-tmp\") pod \"openshift-lws-operator-fd8ccff4c-r7s7d\" (UID: \"3002f8b0-31d8-4b91-a90e-c5d7ea43cfdb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r7s7d" Sep 14 11:05:35.392528 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:05:35.392513 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3002f8b0-31d8-4b91-a90e-c5d7ea43cfdb-tmp\") pod \"openshift-lws-operator-fd8ccff4c-r7s7d\" (UID: \"3002f8b0-31d8-4b91-a90e-c5d7ea43cfdb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r7s7d" Sep 14 11:05:35.400644 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:05:35.400624 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76cqj\" (UniqueName: \"kubernetes.io/projected/3002f8b0-31d8-4b91-a90e-c5d7ea43cfdb-kube-api-access-76cqj\") pod \"openshift-lws-operator-fd8ccff4c-r7s7d\" (UID: \"3002f8b0-31d8-4b91-a90e-c5d7ea43cfdb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r7s7d" Sep 14 11:05:35.508026 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:05:35.507949 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r7s7d" Sep 14 11:05:35.775429 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:05:35.775354 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r7s7d"] Sep 14 11:05:35.778152 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:05:35.778123 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3002f8b0_31d8_4b91_a90e_c5d7ea43cfdb.slice/crio-032ae8f2c091a254493b38f80edac6236c37d4e08c050deb86c2eebd0b63f3ee WatchSource:0}: Error finding container 032ae8f2c091a254493b38f80edac6236c37d4e08c050deb86c2eebd0b63f3ee: Status 404 returned error can't find the container with id 032ae8f2c091a254493b38f80edac6236c37d4e08c050deb86c2eebd0b63f3ee Sep 14 11:05:36.423365 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:05:36.423327 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r7s7d" event={"ID":"3002f8b0-31d8-4b91-a90e-c5d7ea43cfdb","Type":"ContainerStarted","Data":"032ae8f2c091a254493b38f80edac6236c37d4e08c050deb86c2eebd0b63f3ee"} Sep 14 11:05:38.432829 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:05:38.432793 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r7s7d" event={"ID":"3002f8b0-31d8-4b91-a90e-c5d7ea43cfdb","Type":"ContainerStarted","Data":"f71323defcbd92a67a8c6f3b5131a0afec210d2a58a74e2242f6d92c0c522392"} Sep 14 11:05:38.452482 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:05:38.452439 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r7s7d" podStartSLOduration=1.223175206 podStartE2EDuration="3.452424398s" totalImagesPullingTime="2.229249192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:05:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:05:38.451573344 +0000 UTC m=+296.514976254" watchObservedRunningTime="2026-09-14 11:05:38.452424398 +0000 UTC m=+296.515827310" Sep 14 11:05:42.302717 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:05:42.302686 2556 kubelet.go:1648] "Image garbage collection succeeded" Sep 14 11:06:00.305694 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.305664 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx"] Sep 14 11:06:00.308996 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.308977 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx" Sep 14 11:06:00.313042 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.311990 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 14 11:06:00.313042 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.312228 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 14 11:06:00.313042 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.312281 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 14 11:06:00.313042 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.312429 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wrjmp\"" Sep 14 11:06:00.313042 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.312969 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 14 11:06:00.344158 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.344130 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx"] Sep 14 11:06:00.353348 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.353328 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b80f3c5-2662-420e-b7b4-ef154cceee05-tmp\") pod \"opendatahub-operator-controller-manager-6f85645c65-r4zgx\" (UID: \"7b80f3c5-2662-420e-b7b4-ef154cceee05\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx" Sep 14 11:06:00.353443 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.353361 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b80f3c5-2662-420e-b7b4-ef154cceee05-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f85645c65-r4zgx\" (UID: \"7b80f3c5-2662-420e-b7b4-ef154cceee05\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx" Sep 14 11:06:00.353443 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.353401 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b80f3c5-2662-420e-b7b4-ef154cceee05-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f85645c65-r4zgx\" (UID: \"7b80f3c5-2662-420e-b7b4-ef154cceee05\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx" Sep 14 11:06:00.353532 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.353465 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvr8b\" (UniqueName: \"kubernetes.io/projected/7b80f3c5-2662-420e-b7b4-ef154cceee05-kube-api-access-tvr8b\") pod \"opendatahub-operator-controller-manager-6f85645c65-r4zgx\" (UID: \"7b80f3c5-2662-420e-b7b4-ef154cceee05\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx" Sep 14 11:06:00.353532 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.353491 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b80f3c5-2662-420e-b7b4-ef154cceee05-manifests\") pod \"opendatahub-operator-controller-manager-6f85645c65-r4zgx\" (UID: \"7b80f3c5-2662-420e-b7b4-ef154cceee05\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx" Sep 14 11:06:00.454311 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.454282 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvr8b\" (UniqueName: \"kubernetes.io/projected/7b80f3c5-2662-420e-b7b4-ef154cceee05-kube-api-access-tvr8b\") pod \"opendatahub-operator-controller-manager-6f85645c65-r4zgx\" (UID: \"7b80f3c5-2662-420e-b7b4-ef154cceee05\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx" Sep 14 11:06:00.454311 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.454313 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b80f3c5-2662-420e-b7b4-ef154cceee05-manifests\") pod \"opendatahub-operator-controller-manager-6f85645c65-r4zgx\" (UID: \"7b80f3c5-2662-420e-b7b4-ef154cceee05\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx" Sep 14 11:06:00.454521 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.454346 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b80f3c5-2662-420e-b7b4-ef154cceee05-tmp\") pod \"opendatahub-operator-controller-manager-6f85645c65-r4zgx\" (UID: \"7b80f3c5-2662-420e-b7b4-ef154cceee05\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx" Sep 14 11:06:00.454521 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.454372 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b80f3c5-2662-420e-b7b4-ef154cceee05-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f85645c65-r4zgx\" (UID: \"7b80f3c5-2662-420e-b7b4-ef154cceee05\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx" Sep 14 11:06:00.454521 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.454410 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b80f3c5-2662-420e-b7b4-ef154cceee05-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f85645c65-r4zgx\" (UID: \"7b80f3c5-2662-420e-b7b4-ef154cceee05\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx" Sep 14 11:06:00.454751 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.454731 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b80f3c5-2662-420e-b7b4-ef154cceee05-manifests\") pod \"opendatahub-operator-controller-manager-6f85645c65-r4zgx\" (UID: \"7b80f3c5-2662-420e-b7b4-ef154cceee05\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx" Sep 14 11:06:00.456758 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.456739 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b80f3c5-2662-420e-b7b4-ef154cceee05-tmp\") pod \"opendatahub-operator-controller-manager-6f85645c65-r4zgx\" (UID: \"7b80f3c5-2662-420e-b7b4-ef154cceee05\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx" Sep 14 11:06:00.456861 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.456763 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b80f3c5-2662-420e-b7b4-ef154cceee05-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f85645c65-r4zgx\" (UID: \"7b80f3c5-2662-420e-b7b4-ef154cceee05\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx" Sep 14 11:06:00.456861 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.456829 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b80f3c5-2662-420e-b7b4-ef154cceee05-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f85645c65-r4zgx\" (UID: \"7b80f3c5-2662-420e-b7b4-ef154cceee05\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx" Sep 14 11:06:00.472951 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.472929 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvr8b\" (UniqueName: \"kubernetes.io/projected/7b80f3c5-2662-420e-b7b4-ef154cceee05-kube-api-access-tvr8b\") pod \"opendatahub-operator-controller-manager-6f85645c65-r4zgx\" (UID: \"7b80f3c5-2662-420e-b7b4-ef154cceee05\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx" Sep 14 11:06:00.623220 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.623194 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx" Sep 14 11:06:00.906329 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.906305 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx"] Sep 14 11:06:00.908223 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:06:00.908201 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b80f3c5_2662_420e_b7b4_ef154cceee05.slice/crio-11189b0d8562d2e2bfc1d8dcad31937ace4c79f642ad3b27d4e20714d0ea7cc9 WatchSource:0}: Error finding container 11189b0d8562d2e2bfc1d8dcad31937ace4c79f642ad3b27d4e20714d0ea7cc9: Status 404 returned error can't find the container with id 11189b0d8562d2e2bfc1d8dcad31937ace4c79f642ad3b27d4e20714d0ea7cc9 Sep 14 11:06:00.909888 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:00.909872 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 11:06:01.506432 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:01.506392 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx" event={"ID":"7b80f3c5-2662-420e-b7b4-ef154cceee05","Type":"ContainerStarted","Data":"11189b0d8562d2e2bfc1d8dcad31937ace4c79f642ad3b27d4e20714d0ea7cc9"} Sep 14 11:06:04.518672 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:04.518638 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx" event={"ID":"7b80f3c5-2662-420e-b7b4-ef154cceee05","Type":"ContainerStarted","Data":"92a000ec0fb99ae3cf1ad1c2bfcb524a8a57a4b3b0c43a5b04d2038bf9ea2c13"} Sep 14 11:06:05.523638 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:05.523606 2556 generic.go:356] "Generic (PLEG): container finished" podID="7b80f3c5-2662-420e-b7b4-ef154cceee05" containerID="92a000ec0fb99ae3cf1ad1c2bfcb524a8a57a4b3b0c43a5b04d2038bf9ea2c13" exitCode=0 Sep 14 11:06:05.523989 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:05.523700 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx" event={"ID":"7b80f3c5-2662-420e-b7b4-ef154cceee05","Type":"ContainerDied","Data":"92a000ec0fb99ae3cf1ad1c2bfcb524a8a57a4b3b0c43a5b04d2038bf9ea2c13"} Sep 14 11:06:06.529692 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:06.529614 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx" event={"ID":"7b80f3c5-2662-420e-b7b4-ef154cceee05","Type":"ContainerStarted","Data":"9e8e788d8e43181fc535c4240a454400d547d4066a5a19411516a45bb6fb5014"} Sep 14 11:06:06.530028 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:06.529738 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx" Sep 14 11:06:06.568602 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:06.568558 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx" podStartSLOduration=1.813102582 podStartE2EDuration="6.568545127s" totalImagesPullingTime="3.683371686s" totalInitContainerRuntime="1.072070859s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:06:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:06:06.566109785 +0000 UTC m=+324.629512693" watchObservedRunningTime="2026-09-14 11:06:06.568545127 +0000 UTC m=+324.631948038" Sep 14 11:06:17.535299 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:17.535271 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-r4zgx" Sep 14 11:06:18.827840 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:18.827808 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-69cb796bd9-h7xm9"] Sep 14 11:06:18.834697 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:18.834679 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-h7xm9" Sep 14 11:06:18.839927 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:18.839907 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-lvnq9\"" Sep 14 11:06:18.840881 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:18.840857 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 14 11:06:18.840975 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:18.840893 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 14 11:06:18.840975 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:18.840894 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 14 11:06:18.850143 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:18.850122 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-69cb796bd9-h7xm9"] Sep 14 11:06:18.882295 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:18.882270 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/91c62911-b234-42ff-82c3-739cbd1f14c3-metrics-cert\") pod \"lws-controller-manager-69cb796bd9-h7xm9\" (UID: \"91c62911-b234-42ff-82c3-739cbd1f14c3\") " pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-h7xm9" Sep 14 11:06:18.882387 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:18.882303 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/91c62911-b234-42ff-82c3-739cbd1f14c3-manager-config\") pod \"lws-controller-manager-69cb796bd9-h7xm9\" (UID: \"91c62911-b234-42ff-82c3-739cbd1f14c3\") " pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-h7xm9" Sep 14 11:06:18.882387 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:18.882331 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lw85\" (UniqueName: \"kubernetes.io/projected/91c62911-b234-42ff-82c3-739cbd1f14c3-kube-api-access-5lw85\") pod \"lws-controller-manager-69cb796bd9-h7xm9\" (UID: \"91c62911-b234-42ff-82c3-739cbd1f14c3\") " pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-h7xm9" Sep 14 11:06:18.882478 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:18.882396 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91c62911-b234-42ff-82c3-739cbd1f14c3-cert\") pod \"lws-controller-manager-69cb796bd9-h7xm9\" (UID: \"91c62911-b234-42ff-82c3-739cbd1f14c3\") " pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-h7xm9" Sep 14 11:06:18.983120 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:18.983082 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lw85\" (UniqueName: \"kubernetes.io/projected/91c62911-b234-42ff-82c3-739cbd1f14c3-kube-api-access-5lw85\") pod \"lws-controller-manager-69cb796bd9-h7xm9\" (UID: \"91c62911-b234-42ff-82c3-739cbd1f14c3\") " pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-h7xm9" Sep 14 11:06:18.983203 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:18.983127 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91c62911-b234-42ff-82c3-739cbd1f14c3-cert\") pod \"lws-controller-manager-69cb796bd9-h7xm9\" (UID: \"91c62911-b234-42ff-82c3-739cbd1f14c3\") " pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-h7xm9" Sep 14 11:06:18.983203 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:18.983159 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/91c62911-b234-42ff-82c3-739cbd1f14c3-metrics-cert\") pod \"lws-controller-manager-69cb796bd9-h7xm9\" (UID: \"91c62911-b234-42ff-82c3-739cbd1f14c3\") " pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-h7xm9" Sep 14 11:06:18.983332 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:18.983314 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/91c62911-b234-42ff-82c3-739cbd1f14c3-manager-config\") pod \"lws-controller-manager-69cb796bd9-h7xm9\" (UID: \"91c62911-b234-42ff-82c3-739cbd1f14c3\") " pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-h7xm9" Sep 14 11:06:18.983904 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:18.983883 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/91c62911-b234-42ff-82c3-739cbd1f14c3-manager-config\") pod \"lws-controller-manager-69cb796bd9-h7xm9\" (UID: \"91c62911-b234-42ff-82c3-739cbd1f14c3\") " pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-h7xm9" Sep 14 11:06:18.985560 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:18.985531 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/91c62911-b234-42ff-82c3-739cbd1f14c3-metrics-cert\") pod \"lws-controller-manager-69cb796bd9-h7xm9\" (UID: \"91c62911-b234-42ff-82c3-739cbd1f14c3\") " pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-h7xm9" Sep 14 11:06:18.985646 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:18.985563 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91c62911-b234-42ff-82c3-739cbd1f14c3-cert\") pod \"lws-controller-manager-69cb796bd9-h7xm9\" (UID: \"91c62911-b234-42ff-82c3-739cbd1f14c3\") " pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-h7xm9" Sep 14 11:06:18.992268 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:18.992248 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lw85\" (UniqueName: \"kubernetes.io/projected/91c62911-b234-42ff-82c3-739cbd1f14c3-kube-api-access-5lw85\") pod \"lws-controller-manager-69cb796bd9-h7xm9\" (UID: \"91c62911-b234-42ff-82c3-739cbd1f14c3\") " pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-h7xm9" Sep 14 11:06:19.143586 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:19.143553 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-h7xm9" Sep 14 11:06:19.420597 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:19.420414 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-69cb796bd9-h7xm9"] Sep 14 11:06:19.423125 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:06:19.423099 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c62911_b234_42ff_82c3_739cbd1f14c3.slice/crio-a370ba8bd239b56b851199a89d29c08281d1bb31da087b335947b84b87986f48 WatchSource:0}: Error finding container a370ba8bd239b56b851199a89d29c08281d1bb31da087b335947b84b87986f48: Status 404 returned error can't find the container with id a370ba8bd239b56b851199a89d29c08281d1bb31da087b335947b84b87986f48 Sep 14 11:06:19.572764 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:19.572733 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-h7xm9" event={"ID":"91c62911-b234-42ff-82c3-739cbd1f14c3","Type":"ContainerStarted","Data":"a370ba8bd239b56b851199a89d29c08281d1bb31da087b335947b84b87986f48"} Sep 14 11:06:22.586608 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:22.586521 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-h7xm9" event={"ID":"91c62911-b234-42ff-82c3-739cbd1f14c3","Type":"ContainerStarted","Data":"dab39f5c22fbceff7b90c058296fd4aa013303665e17ef82513973b7f33ce4a4"} Sep 14 11:06:22.587001 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:22.586670 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-h7xm9" Sep 14 11:06:22.611279 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:22.611237 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-h7xm9" podStartSLOduration=1.913374387 podStartE2EDuration="4.611224362s" totalImagesPullingTime="2.697849975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:06:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:06:22.609808888 +0000 UTC m=+340.673211798" watchObservedRunningTime="2026-09-14 11:06:22.611224362 +0000 UTC m=+340.674627273" Sep 14 11:06:28.919700 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:28.919661 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248"] Sep 14 11:06:28.921723 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:28.921701 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:28.929842 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:28.929823 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 14 11:06:28.929953 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:28.929840 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 14 11:06:28.930682 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:28.930666 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-kwhfk\"" Sep 14 11:06:28.930765 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:28.930666 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 14 11:06:28.946722 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:28.946701 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248"] Sep 14 11:06:29.033787 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.033755 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt"] Sep 14 11:06:29.035746 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.035729 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.054447 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.054428 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.054549 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.054455 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.054549 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.054475 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.054549 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.054495 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.054671 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.054542 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b090c68-efa5-4375-81be-6e12fe34a8aa-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.054671 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.054571 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b090c68-efa5-4375-81be-6e12fe34a8aa-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.054671 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.054612 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.054765 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.054672 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b090c68-efa5-4375-81be-6e12fe34a8aa-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.054765 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.054724 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2cr6\" (UniqueName: \"kubernetes.io/projected/7b090c68-efa5-4375-81be-6e12fe34a8aa-kube-api-access-r2cr6\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.068784 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.068758 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt"] Sep 14 11:06:29.155632 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.155591 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c85e1d-391a-4ff7-a402-9739d226bd4b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.155632 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.155634 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c85e1d-391a-4ff7-a402-9739d226bd4b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.155848 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.155687 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c85e1d-391a-4ff7-a402-9739d226bd4b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.155848 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.155751 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c85e1d-391a-4ff7-a402-9739d226bd4b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.155848 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.155779 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c85e1d-391a-4ff7-a402-9739d226bd4b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.155848 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.155804 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c85e1d-391a-4ff7-a402-9739d226bd4b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.156028 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.155865 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.156028 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.155886 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.156028 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.155903 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.156028 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.155930 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c85e1d-391a-4ff7-a402-9739d226bd4b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.156028 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.155957 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.156028 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.155999 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b090c68-efa5-4375-81be-6e12fe34a8aa-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.156347 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.156033 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b090c68-efa5-4375-81be-6e12fe34a8aa-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.156347 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.156064 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.156347 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.156207 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b090c68-efa5-4375-81be-6e12fe34a8aa-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.156347 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.156258 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c85e1d-391a-4ff7-a402-9739d226bd4b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.156347 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.156324 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2cr6\" (UniqueName: \"kubernetes.io/projected/7b090c68-efa5-4375-81be-6e12fe34a8aa-kube-api-access-r2cr6\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.156587 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.156348 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsh9z\" (UniqueName: \"kubernetes.io/projected/f6c85e1d-391a-4ff7-a402-9739d226bd4b-kube-api-access-dsh9z\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.156587 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.156374 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.156587 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.156552 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.156587 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.156569 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.156745 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.156640 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.156920 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.156896 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b090c68-efa5-4375-81be-6e12fe34a8aa-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.158579 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.158550 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.158579 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.158565 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b090c68-efa5-4375-81be-6e12fe34a8aa-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.167425 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.167399 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b090c68-efa5-4375-81be-6e12fe34a8aa-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.167806 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.167786 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2cr6\" (UniqueName: \"kubernetes.io/projected/7b090c68-efa5-4375-81be-6e12fe34a8aa-kube-api-access-r2cr6\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc25248\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.231003 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.230915 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:29.256891 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.256860 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsh9z\" (UniqueName: \"kubernetes.io/projected/f6c85e1d-391a-4ff7-a402-9739d226bd4b-kube-api-access-dsh9z\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.257032 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.256905 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c85e1d-391a-4ff7-a402-9739d226bd4b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.257032 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.256937 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c85e1d-391a-4ff7-a402-9739d226bd4b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.257032 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.256973 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c85e1d-391a-4ff7-a402-9739d226bd4b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.257032 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.257002 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c85e1d-391a-4ff7-a402-9739d226bd4b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.257032 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.257025 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c85e1d-391a-4ff7-a402-9739d226bd4b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.257282 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.257047 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c85e1d-391a-4ff7-a402-9739d226bd4b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.257352 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.257317 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c85e1d-391a-4ff7-a402-9739d226bd4b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.257352 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.257330 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c85e1d-391a-4ff7-a402-9739d226bd4b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.257542 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.257375 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c85e1d-391a-4ff7-a402-9739d226bd4b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.257542 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.257412 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c85e1d-391a-4ff7-a402-9739d226bd4b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.257542 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.257419 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c85e1d-391a-4ff7-a402-9739d226bd4b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.257810 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.257784 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c85e1d-391a-4ff7-a402-9739d226bd4b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.258062 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.258043 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c85e1d-391a-4ff7-a402-9739d226bd4b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.259442 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.259425 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c85e1d-391a-4ff7-a402-9739d226bd4b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.259674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.259656 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c85e1d-391a-4ff7-a402-9739d226bd4b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.272824 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.272797 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c85e1d-391a-4ff7-a402-9739d226bd4b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.282170 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.282147 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsh9z\" (UniqueName: \"kubernetes.io/projected/f6c85e1d-391a-4ff7-a402-9739d226bd4b-kube-api-access-dsh9z\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt\" (UID: \"f6c85e1d-391a-4ff7-a402-9739d226bd4b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.345587 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.345556 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:29.512261 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.512190 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248"] Sep 14 11:06:29.515480 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:06:29.515451 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b090c68_efa5_4375_81be_6e12fe34a8aa.slice/crio-25ad628d56f6e5a7045468d4ec5f0d1f32359c870fb248b1cff4c60d9a6badfb WatchSource:0}: Error finding container 25ad628d56f6e5a7045468d4ec5f0d1f32359c870fb248b1cff4c60d9a6badfb: Status 404 returned error can't find the container with id 25ad628d56f6e5a7045468d4ec5f0d1f32359c870fb248b1cff4c60d9a6badfb Sep 14 11:06:29.611922 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.611888 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" event={"ID":"7b090c68-efa5-4375-81be-6e12fe34a8aa","Type":"ContainerStarted","Data":"25ad628d56f6e5a7045468d4ec5f0d1f32359c870fb248b1cff4c60d9a6badfb"} Sep 14 11:06:29.619305 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:29.619280 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt"] Sep 14 11:06:29.622237 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:06:29.622215 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c85e1d_391a_4ff7_a402_9739d226bd4b.slice/crio-a80648102fbc7f07b55d5bc83caf288754d677a0eb9aaae8c7542d172edbc19a WatchSource:0}: Error finding container a80648102fbc7f07b55d5bc83caf288754d677a0eb9aaae8c7542d172edbc19a: Status 404 returned error can't find the container with id a80648102fbc7f07b55d5bc83caf288754d677a0eb9aaae8c7542d172edbc19a Sep 14 11:06:30.619492 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:30.619451 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" event={"ID":"f6c85e1d-391a-4ff7-a402-9739d226bd4b","Type":"ContainerStarted","Data":"a80648102fbc7f07b55d5bc83caf288754d677a0eb9aaae8c7542d172edbc19a"} Sep 14 11:06:32.061199 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:32.061151 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:06:32.061491 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:32.061234 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:06:32.061491 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:32.061285 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:06:32.067714 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:32.067683 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:06:32.067819 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:32.067748 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:06:32.067819 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:32.067778 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:06:32.630848 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:32.630797 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" event={"ID":"f6c85e1d-391a-4ff7-a402-9739d226bd4b","Type":"ContainerStarted","Data":"c6c04e1b65b1855c02766a698c4a8c87a60f49500b83ad7c732c740c53e6bc67"} Sep 14 11:06:32.632471 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:32.632441 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" event={"ID":"7b090c68-efa5-4375-81be-6e12fe34a8aa","Type":"ContainerStarted","Data":"24ae52921f88942b3657aef4f5dc3029aff723ec9510ef9f8aae5b8474e2b274"} Sep 14 11:06:32.656776 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:32.656724 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" podStartSLOduration=1.220062491 podStartE2EDuration="3.656707984s" totalImagesPullingTime="2.436645493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:06:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:06:32.654702712 +0000 UTC m=+350.718105623" watchObservedRunningTime="2026-09-14 11:06:32.656707984 +0000 UTC m=+350.720110898" Sep 14 11:06:32.683839 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:32.683795 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" podStartSLOduration=2.133765926 podStartE2EDuration="4.683780257s" totalImagesPullingTime="2.550014331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:06:32.683395829 +0000 UTC m=+350.746798741" watchObservedRunningTime="2026-09-14 11:06:32.683780257 +0000 UTC m=+350.747183169" Sep 14 11:06:33.231506 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:33.231479 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:33.232686 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:33.232655 2556 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bc25248 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.16:15021/healthz/ready\": dial tcp 10.132.0.16:15021: connect: connection refused" start-of-body= Sep 14 11:06:33.232809 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:33.232700 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" podUID="7b090c68-efa5-4375-81be-6e12fe34a8aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.16:15021/healthz/ready\": dial tcp 10.132.0.16:15021: connect: connection refused" Sep 14 11:06:33.346440 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:33.346413 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:33.350359 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:33.350338 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:33.591714 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:33.591645 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-h7xm9" Sep 14 11:06:33.636940 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:33.636915 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:33.637485 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:33.637468 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt" Sep 14 11:06:33.767888 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:33.767863 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248"] Sep 14 11:06:34.231955 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:34.231920 2556 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bc25248 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.16:15021/healthz/ready\": dial tcp 10.132.0.16:15021: connect: connection refused" start-of-body= Sep 14 11:06:34.232349 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:34.231983 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" podUID="7b090c68-efa5-4375-81be-6e12fe34a8aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.16:15021/healthz/ready\": dial tcp 10.132.0.16:15021: connect: connection refused" Sep 14 11:06:35.231796 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:35.231713 2556 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bc25248 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.16:15021/healthz/ready\": dial tcp 10.132.0.16:15021: connect: connection refused" start-of-body= Sep 14 11:06:35.231796 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:35.231771 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" podUID="7b090c68-efa5-4375-81be-6e12fe34a8aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.16:15021/healthz/ready\": dial tcp 10.132.0.16:15021: connect: connection refused" Sep 14 11:06:35.645643 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:35.645591 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" podUID="7b090c68-efa5-4375-81be-6e12fe34a8aa" containerName="istio-proxy" containerID="cri-o://24ae52921f88942b3657aef4f5dc3029aff723ec9510ef9f8aae5b8474e2b274" gracePeriod=30 Sep 14 11:06:40.878878 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:40.878857 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:40.948980 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:40.948924 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-workload-socket\") pod \"7b090c68-efa5-4375-81be-6e12fe34a8aa\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " Sep 14 11:06:40.948980 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:40.948949 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-istio-envoy\") pod \"7b090c68-efa5-4375-81be-6e12fe34a8aa\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " Sep 14 11:06:40.948980 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:40.948973 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-istio-data\") pod \"7b090c68-efa5-4375-81be-6e12fe34a8aa\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " Sep 14 11:06:40.949214 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:40.948998 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b090c68-efa5-4375-81be-6e12fe34a8aa-istio-podinfo\") pod \"7b090c68-efa5-4375-81be-6e12fe34a8aa\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " Sep 14 11:06:40.949214 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:40.949024 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2cr6\" (UniqueName: \"kubernetes.io/projected/7b090c68-efa5-4375-81be-6e12fe34a8aa-kube-api-access-r2cr6\") pod \"7b090c68-efa5-4375-81be-6e12fe34a8aa\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " Sep 14 11:06:40.949214 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:40.949051 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-workload-certs\") pod \"7b090c68-efa5-4375-81be-6e12fe34a8aa\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " Sep 14 11:06:40.949214 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:40.949121 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-credential-socket\") pod \"7b090c68-efa5-4375-81be-6e12fe34a8aa\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " Sep 14 11:06:40.949214 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:40.949164 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b090c68-efa5-4375-81be-6e12fe34a8aa-istio-token\") pod \"7b090c68-efa5-4375-81be-6e12fe34a8aa\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " Sep 14 11:06:40.949214 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:40.949194 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b090c68-efa5-4375-81be-6e12fe34a8aa-istiod-ca-cert\") pod \"7b090c68-efa5-4375-81be-6e12fe34a8aa\" (UID: \"7b090c68-efa5-4375-81be-6e12fe34a8aa\") " Sep 14 11:06:40.949525 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:40.949246 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-istio-data" (OuterVolumeSpecName: "istio-data") pod "7b090c68-efa5-4375-81be-6e12fe34a8aa" (UID: "7b090c68-efa5-4375-81be-6e12fe34a8aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:06:40.949525 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:40.949453 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7b090c68-efa5-4375-81be-6e12fe34a8aa" (UID: "7b090c68-efa5-4375-81be-6e12fe34a8aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:06:40.949525 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:40.949453 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7b090c68-efa5-4375-81be-6e12fe34a8aa" (UID: "7b090c68-efa5-4375-81be-6e12fe34a8aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:06:40.949525 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:40.949514 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:06:40.949725 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:40.949521 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7b090c68-efa5-4375-81be-6e12fe34a8aa" (UID: "7b090c68-efa5-4375-81be-6e12fe34a8aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:06:40.949725 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:40.949591 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b090c68-efa5-4375-81be-6e12fe34a8aa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7b090c68-efa5-4375-81be-6e12fe34a8aa" (UID: "7b090c68-efa5-4375-81be-6e12fe34a8aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:06:40.951284 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:40.951256 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b090c68-efa5-4375-81be-6e12fe34a8aa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7b090c68-efa5-4375-81be-6e12fe34a8aa" (UID: "7b090c68-efa5-4375-81be-6e12fe34a8aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 11:06:40.951380 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:40.951303 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7b090c68-efa5-4375-81be-6e12fe34a8aa" (UID: "7b090c68-efa5-4375-81be-6e12fe34a8aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:06:40.951380 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:40.951336 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b090c68-efa5-4375-81be-6e12fe34a8aa-istio-token" (OuterVolumeSpecName: "istio-token") pod "7b090c68-efa5-4375-81be-6e12fe34a8aa" (UID: "7b090c68-efa5-4375-81be-6e12fe34a8aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:06:40.951451 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:40.951396 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b090c68-efa5-4375-81be-6e12fe34a8aa-kube-api-access-r2cr6" (OuterVolumeSpecName: "kube-api-access-r2cr6") pod "7b090c68-efa5-4375-81be-6e12fe34a8aa" (UID: "7b090c68-efa5-4375-81be-6e12fe34a8aa"). InnerVolumeSpecName "kube-api-access-r2cr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:06:41.050792 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:41.050756 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b090c68-efa5-4375-81be-6e12fe34a8aa-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:06:41.050792 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:41.050790 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2cr6\" (UniqueName: \"kubernetes.io/projected/7b090c68-efa5-4375-81be-6e12fe34a8aa-kube-api-access-r2cr6\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:06:41.050792 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:41.050802 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:06:41.050964 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:41.050810 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:06:41.050964 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:41.050818 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b090c68-efa5-4375-81be-6e12fe34a8aa-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:06:41.050964 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:41.050826 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b090c68-efa5-4375-81be-6e12fe34a8aa-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:06:41.050964 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:41.050833 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:06:41.050964 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:41.050841 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b090c68-efa5-4375-81be-6e12fe34a8aa-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:06:41.668808 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:41.668771 2556 generic.go:356] "Generic (PLEG): container finished" podID="7b090c68-efa5-4375-81be-6e12fe34a8aa" containerID="24ae52921f88942b3657aef4f5dc3029aff723ec9510ef9f8aae5b8474e2b274" exitCode=0 Sep 14 11:06:41.668960 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:41.668835 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" Sep 14 11:06:41.668960 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:41.668861 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" event={"ID":"7b090c68-efa5-4375-81be-6e12fe34a8aa","Type":"ContainerDied","Data":"24ae52921f88942b3657aef4f5dc3029aff723ec9510ef9f8aae5b8474e2b274"} Sep 14 11:06:41.668960 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:41.668916 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248" event={"ID":"7b090c68-efa5-4375-81be-6e12fe34a8aa","Type":"ContainerDied","Data":"25ad628d56f6e5a7045468d4ec5f0d1f32359c870fb248b1cff4c60d9a6badfb"} Sep 14 11:06:41.668960 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:41.668938 2556 scope.go:117] "RemoveContainer" containerID="24ae52921f88942b3657aef4f5dc3029aff723ec9510ef9f8aae5b8474e2b274" Sep 14 11:06:41.677672 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:41.677652 2556 scope.go:117] "RemoveContainer" containerID="24ae52921f88942b3657aef4f5dc3029aff723ec9510ef9f8aae5b8474e2b274" Sep 14 11:06:41.677925 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:06:41.677905 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24ae52921f88942b3657aef4f5dc3029aff723ec9510ef9f8aae5b8474e2b274\": container with ID starting with 24ae52921f88942b3657aef4f5dc3029aff723ec9510ef9f8aae5b8474e2b274 not found: ID does not exist" containerID="24ae52921f88942b3657aef4f5dc3029aff723ec9510ef9f8aae5b8474e2b274" Sep 14 11:06:41.677972 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:41.677934 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24ae52921f88942b3657aef4f5dc3029aff723ec9510ef9f8aae5b8474e2b274"} err="failed to get container status \"24ae52921f88942b3657aef4f5dc3029aff723ec9510ef9f8aae5b8474e2b274\": rpc error: code = NotFound desc = could not find container \"24ae52921f88942b3657aef4f5dc3029aff723ec9510ef9f8aae5b8474e2b274\": container with ID starting with 24ae52921f88942b3657aef4f5dc3029aff723ec9510ef9f8aae5b8474e2b274 not found: ID does not exist" Sep 14 11:06:41.713768 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:41.713744 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248"] Sep 14 11:06:41.739948 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:41.739925 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc25248"] Sep 14 11:06:42.381515 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:42.381479 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b090c68-efa5-4375-81be-6e12fe34a8aa" path="/var/lib/kubelet/pods/7b090c68-efa5-4375-81be-6e12fe34a8aa/volumes" Sep 14 11:06:44.544633 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:44.544599 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pgqgl"] Sep 14 11:06:44.544953 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:44.544853 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b090c68-efa5-4375-81be-6e12fe34a8aa" containerName="istio-proxy" Sep 14 11:06:44.544953 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:44.544864 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b090c68-efa5-4375-81be-6e12fe34a8aa" containerName="istio-proxy" Sep 14 11:06:44.544953 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:44.544903 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b090c68-efa5-4375-81be-6e12fe34a8aa" containerName="istio-proxy" Sep 14 11:06:44.547876 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:44.547858 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgqgl" Sep 14 11:06:44.555210 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:44.555189 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-phbcx\"" Sep 14 11:06:44.555304 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:44.555232 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 14 11:06:44.644353 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:44.644316 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pgqgl"] Sep 14 11:06:44.675081 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:44.675046 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7de2f60-4aa0-4b18-badf-f63301c93d4d-cert\") pod \"kserve-controller-manager-d6f4b65b8-pgqgl\" (UID: \"e7de2f60-4aa0-4b18-badf-f63301c93d4d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgqgl" Sep 14 11:06:44.675248 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:44.675161 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xts8\" (UniqueName: \"kubernetes.io/projected/e7de2f60-4aa0-4b18-badf-f63301c93d4d-kube-api-access-7xts8\") pod \"kserve-controller-manager-d6f4b65b8-pgqgl\" (UID: \"e7de2f60-4aa0-4b18-badf-f63301c93d4d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgqgl" Sep 14 11:06:44.706596 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:44.706567 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zcmgx"] Sep 14 11:06:44.708536 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:44.708519 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zcmgx" Sep 14 11:06:44.751941 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:44.751914 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-ts5g8\"" Sep 14 11:06:44.759454 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:44.759434 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 14 11:06:44.775914 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:44.775887 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xts8\" (UniqueName: \"kubernetes.io/projected/e7de2f60-4aa0-4b18-badf-f63301c93d4d-kube-api-access-7xts8\") pod \"kserve-controller-manager-d6f4b65b8-pgqgl\" (UID: \"e7de2f60-4aa0-4b18-badf-f63301c93d4d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgqgl" Sep 14 11:06:44.776005 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:44.775932 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7de2f60-4aa0-4b18-badf-f63301c93d4d-cert\") pod \"kserve-controller-manager-d6f4b65b8-pgqgl\" (UID: \"e7de2f60-4aa0-4b18-badf-f63301c93d4d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgqgl" Sep 14 11:06:44.778300 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:44.778279 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7de2f60-4aa0-4b18-badf-f63301c93d4d-cert\") pod \"kserve-controller-manager-d6f4b65b8-pgqgl\" (UID: \"e7de2f60-4aa0-4b18-badf-f63301c93d4d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgqgl" Sep 14 11:06:44.843364 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:44.843290 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zcmgx"] Sep 14 11:06:44.868414 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:44.868387 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xts8\" (UniqueName: \"kubernetes.io/projected/e7de2f60-4aa0-4b18-badf-f63301c93d4d-kube-api-access-7xts8\") pod \"kserve-controller-manager-d6f4b65b8-pgqgl\" (UID: \"e7de2f60-4aa0-4b18-badf-f63301c93d4d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgqgl" Sep 14 11:06:44.876283 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:44.876254 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb948054-387f-4074-94e5-c798756fe277-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zcmgx\" (UID: \"eb948054-387f-4074-94e5-c798756fe277\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zcmgx" Sep 14 11:06:44.876392 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:44.876334 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr6lj\" (UniqueName: \"kubernetes.io/projected/eb948054-387f-4074-94e5-c798756fe277-kube-api-access-fr6lj\") pod \"llmisvc-controller-manager-777c6bd7d5-zcmgx\" (UID: \"eb948054-387f-4074-94e5-c798756fe277\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zcmgx" Sep 14 11:06:44.977527 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:44.977495 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb948054-387f-4074-94e5-c798756fe277-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zcmgx\" (UID: \"eb948054-387f-4074-94e5-c798756fe277\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zcmgx" Sep 14 11:06:44.977669 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:44.977552 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr6lj\" (UniqueName: \"kubernetes.io/projected/eb948054-387f-4074-94e5-c798756fe277-kube-api-access-fr6lj\") pod \"llmisvc-controller-manager-777c6bd7d5-zcmgx\" (UID: \"eb948054-387f-4074-94e5-c798756fe277\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zcmgx" Sep 14 11:06:44.979950 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:44.979928 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb948054-387f-4074-94e5-c798756fe277-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zcmgx\" (UID: \"eb948054-387f-4074-94e5-c798756fe277\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zcmgx" Sep 14 11:06:45.024110 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:45.024062 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr6lj\" (UniqueName: \"kubernetes.io/projected/eb948054-387f-4074-94e5-c798756fe277-kube-api-access-fr6lj\") pod \"llmisvc-controller-manager-777c6bd7d5-zcmgx\" (UID: \"eb948054-387f-4074-94e5-c798756fe277\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zcmgx" Sep 14 11:06:45.157596 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:45.157574 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgqgl" Sep 14 11:06:45.317312 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:45.317270 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zcmgx" Sep 14 11:06:45.459056 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:45.458953 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pgqgl"] Sep 14 11:06:45.461520 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:06:45.461497 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7de2f60_4aa0_4b18_badf_f63301c93d4d.slice/crio-09735c04bacceb072bf8ad76565b29a2845c46ad8ce27c8f162735a8f3d1dbe1 WatchSource:0}: Error finding container 09735c04bacceb072bf8ad76565b29a2845c46ad8ce27c8f162735a8f3d1dbe1: Status 404 returned error can't find the container with id 09735c04bacceb072bf8ad76565b29a2845c46ad8ce27c8f162735a8f3d1dbe1 Sep 14 11:06:45.611581 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:45.611559 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zcmgx"] Sep 14 11:06:45.613487 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:06:45.613461 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podeb948054_387f_4074_94e5_c798756fe277.slice/crio-af4ef235ea4a1d809054e3d5ef54a81899d0d8b413ae83fb630a59febc61d148 WatchSource:0}: Error finding container af4ef235ea4a1d809054e3d5ef54a81899d0d8b413ae83fb630a59febc61d148: Status 404 returned error can't find the container with id af4ef235ea4a1d809054e3d5ef54a81899d0d8b413ae83fb630a59febc61d148 Sep 14 11:06:45.684134 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:45.684110 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgqgl" event={"ID":"e7de2f60-4aa0-4b18-badf-f63301c93d4d","Type":"ContainerStarted","Data":"09735c04bacceb072bf8ad76565b29a2845c46ad8ce27c8f162735a8f3d1dbe1"} Sep 14 11:06:45.685235 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:45.685214 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zcmgx" event={"ID":"eb948054-387f-4074-94e5-c798756fe277","Type":"ContainerStarted","Data":"af4ef235ea4a1d809054e3d5ef54a81899d0d8b413ae83fb630a59febc61d148"} Sep 14 11:06:49.707156 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:49.707123 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgqgl" event={"ID":"e7de2f60-4aa0-4b18-badf-f63301c93d4d","Type":"ContainerStarted","Data":"55129f786eb7ea5ad7fee6c9fa59b01a0bce7e8c7c4b4b2159c389c0e874180c"} Sep 14 11:06:49.707557 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:49.707208 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgqgl" Sep 14 11:06:49.708811 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:49.708772 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zcmgx" event={"ID":"eb948054-387f-4074-94e5-c798756fe277","Type":"ContainerStarted","Data":"c674e1d0423a10b4f8eb91c18188917814c56ca55d192eefc94c4c089eac042d"} Sep 14 11:06:49.708958 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:49.708908 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zcmgx" Sep 14 11:06:49.746261 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:49.746219 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgqgl" podStartSLOduration=2.264258797 podStartE2EDuration="5.746207264s" totalImagesPullingTime="3.481948467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:06:49.746041573 +0000 UTC m=+367.809444485" watchObservedRunningTime="2026-09-14 11:06:49.746207264 +0000 UTC m=+367.809610176" Sep 14 11:06:49.775449 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:06:49.775406 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zcmgx" podStartSLOduration=2.445370799 podStartE2EDuration="5.775396621s" totalImagesPullingTime="3.330025822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:06:49.775255133 +0000 UTC m=+367.838658043" watchObservedRunningTime="2026-09-14 11:06:49.775396621 +0000 UTC m=+367.838799531" Sep 14 11:07:20.715120 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:07:20.715074 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zcmgx" Sep 14 11:07:20.717972 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:07:20.717953 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgqgl" Sep 14 11:08:18.022374 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:18.022292 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6v8f"] Sep 14 11:08:18.025519 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:18.025502 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6v8f" Sep 14 11:08:18.028150 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:18.028130 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 14 11:08:18.028277 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:18.028132 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-25c6l\"" Sep 14 11:08:18.028277 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:18.028165 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 14 11:08:18.045146 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:18.045125 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6v8f"] Sep 14 11:08:18.132857 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:18.132825 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5btpf\" (UniqueName: \"kubernetes.io/projected/c630b5a2-2f5a-4d78-afac-eeadc2062d1e-kube-api-access-5btpf\") pod \"kuadrant-operator-controller-manager-76f8f686bf-q6v8f\" (UID: \"c630b5a2-2f5a-4d78-afac-eeadc2062d1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6v8f" Sep 14 11:08:18.132994 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:18.132875 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c630b5a2-2f5a-4d78-afac-eeadc2062d1e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-q6v8f\" (UID: \"c630b5a2-2f5a-4d78-afac-eeadc2062d1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6v8f" Sep 14 11:08:18.233493 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:18.233465 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c630b5a2-2f5a-4d78-afac-eeadc2062d1e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-q6v8f\" (UID: \"c630b5a2-2f5a-4d78-afac-eeadc2062d1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6v8f" Sep 14 11:08:18.233622 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:18.233517 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5btpf\" (UniqueName: \"kubernetes.io/projected/c630b5a2-2f5a-4d78-afac-eeadc2062d1e-kube-api-access-5btpf\") pod \"kuadrant-operator-controller-manager-76f8f686bf-q6v8f\" (UID: \"c630b5a2-2f5a-4d78-afac-eeadc2062d1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6v8f" Sep 14 11:08:18.233859 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:18.233838 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c630b5a2-2f5a-4d78-afac-eeadc2062d1e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-q6v8f\" (UID: \"c630b5a2-2f5a-4d78-afac-eeadc2062d1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6v8f" Sep 14 11:08:18.242762 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:18.242739 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5btpf\" (UniqueName: \"kubernetes.io/projected/c630b5a2-2f5a-4d78-afac-eeadc2062d1e-kube-api-access-5btpf\") pod \"kuadrant-operator-controller-manager-76f8f686bf-q6v8f\" (UID: \"c630b5a2-2f5a-4d78-afac-eeadc2062d1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6v8f" Sep 14 11:08:18.335518 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:18.335465 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6v8f" Sep 14 11:08:18.612021 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:18.611993 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6v8f"] Sep 14 11:08:18.616327 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:08:18.616296 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc630b5a2_2f5a_4d78_afac_eeadc2062d1e.slice/crio-16c771e348b063f60e1f704a43a7cd06db05ba7053b7bbc99c7c8cbceb1e725a WatchSource:0}: Error finding container 16c771e348b063f60e1f704a43a7cd06db05ba7053b7bbc99c7c8cbceb1e725a: Status 404 returned error can't find the container with id 16c771e348b063f60e1f704a43a7cd06db05ba7053b7bbc99c7c8cbceb1e725a Sep 14 11:08:19.054840 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:19.054758 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6v8f" event={"ID":"c630b5a2-2f5a-4d78-afac-eeadc2062d1e","Type":"ContainerStarted","Data":"16c771e348b063f60e1f704a43a7cd06db05ba7053b7bbc99c7c8cbceb1e725a"} Sep 14 11:08:29.096415 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:29.096378 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6v8f" event={"ID":"c630b5a2-2f5a-4d78-afac-eeadc2062d1e","Type":"ContainerStarted","Data":"66ed933d6fb141e1b11c6b4c638a745ada316d0999e2698816441c2a807abac3"} Sep 14 11:08:29.096904 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:29.096489 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6v8f" Sep 14 11:08:29.118813 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:29.118766 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6v8f" podStartSLOduration=2.323834122 podStartE2EDuration="12.118753933s" totalImagesPullingTime="9.794919811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:08:29.117385899 +0000 UTC m=+467.180788809" watchObservedRunningTime="2026-09-14 11:08:29.118753933 +0000 UTC m=+467.182156845" Sep 14 11:08:40.102988 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:40.102952 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6v8f" Sep 14 11:08:42.093946 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:42.093916 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6v8f"] Sep 14 11:08:42.094331 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:42.094148 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6v8f" podUID="c630b5a2-2f5a-4d78-afac-eeadc2062d1e" containerName="manager" containerID="cri-o://66ed933d6fb141e1b11c6b4c638a745ada316d0999e2698816441c2a807abac3" gracePeriod=2 Sep 14 11:08:42.118239 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:42.118213 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6v8f"] Sep 14 11:08:42.123978 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:08:42.123945 2556 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"c630b5a2-2f5a-4d78-afac-eeadc2062d1e\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-q6v8f\": pods \"kuadrant-operator-controller-manager-76f8f686bf-q6v8f\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6v8f" Sep 14 11:08:42.171342 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:42.171317 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xlh7s"] Sep 14 11:08:42.171586 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:42.171574 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c630b5a2-2f5a-4d78-afac-eeadc2062d1e" containerName="manager" Sep 14 11:08:42.171628 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:42.171588 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c630b5a2-2f5a-4d78-afac-eeadc2062d1e" containerName="manager" Sep 14 11:08:42.171664 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:42.171638 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c630b5a2-2f5a-4d78-afac-eeadc2062d1e" containerName="manager" Sep 14 11:08:42.174255 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:42.174236 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xlh7s" Sep 14 11:08:42.198571 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:42.198545 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xlh7s"] Sep 14 11:08:42.222711 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:08:42.222679 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-q6v8f\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="c630b5a2-2f5a-4d78-afac-eeadc2062d1e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6v8f" Sep 14 11:08:42.308853 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:42.308824 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j5gv\" (UniqueName: \"kubernetes.io/projected/dab728de-1b9b-42b5-9662-82db6865f2a4-kube-api-access-4j5gv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xlh7s\" (UID: \"dab728de-1b9b-42b5-9662-82db6865f2a4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xlh7s" Sep 14 11:08:42.308995 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:42.308876 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dab728de-1b9b-42b5-9662-82db6865f2a4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xlh7s\" (UID: \"dab728de-1b9b-42b5-9662-82db6865f2a4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xlh7s" Sep 14 11:08:42.326775 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:42.326757 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6v8f" Sep 14 11:08:42.329239 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:08:42.329215 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-q6v8f\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="c630b5a2-2f5a-4d78-afac-eeadc2062d1e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6v8f" Sep 14 11:08:42.379812 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:08:42.379774 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-q6v8f\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="c630b5a2-2f5a-4d78-afac-eeadc2062d1e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6v8f" Sep 14 11:08:42.409138 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:42.409116 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5btpf\" (UniqueName: \"kubernetes.io/projected/c630b5a2-2f5a-4d78-afac-eeadc2062d1e-kube-api-access-5btpf\") pod \"c630b5a2-2f5a-4d78-afac-eeadc2062d1e\" (UID: \"c630b5a2-2f5a-4d78-afac-eeadc2062d1e\") " Sep 14 11:08:42.409238 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:42.409164 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c630b5a2-2f5a-4d78-afac-eeadc2062d1e-extensions-socket-volume\") pod \"c630b5a2-2f5a-4d78-afac-eeadc2062d1e\" (UID: \"c630b5a2-2f5a-4d78-afac-eeadc2062d1e\") " Sep 14 11:08:42.409302 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:42.409261 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dab728de-1b9b-42b5-9662-82db6865f2a4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xlh7s\" (UID: \"dab728de-1b9b-42b5-9662-82db6865f2a4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xlh7s" Sep 14 11:08:42.409357 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:42.409327 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j5gv\" (UniqueName: \"kubernetes.io/projected/dab728de-1b9b-42b5-9662-82db6865f2a4-kube-api-access-4j5gv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xlh7s\" (UID: \"dab728de-1b9b-42b5-9662-82db6865f2a4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xlh7s" Sep 14 11:08:42.409640 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:42.409609 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dab728de-1b9b-42b5-9662-82db6865f2a4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xlh7s\" (UID: \"dab728de-1b9b-42b5-9662-82db6865f2a4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xlh7s" Sep 14 11:08:42.409640 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:42.409626 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c630b5a2-2f5a-4d78-afac-eeadc2062d1e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c630b5a2-2f5a-4d78-afac-eeadc2062d1e" (UID: "c630b5a2-2f5a-4d78-afac-eeadc2062d1e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:08:42.411099 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:42.411070 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c630b5a2-2f5a-4d78-afac-eeadc2062d1e-kube-api-access-5btpf" (OuterVolumeSpecName: "kube-api-access-5btpf") pod "c630b5a2-2f5a-4d78-afac-eeadc2062d1e" (UID: "c630b5a2-2f5a-4d78-afac-eeadc2062d1e"). InnerVolumeSpecName "kube-api-access-5btpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:08:42.422713 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:42.422696 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j5gv\" (UniqueName: \"kubernetes.io/projected/dab728de-1b9b-42b5-9662-82db6865f2a4-kube-api-access-4j5gv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xlh7s\" (UID: \"dab728de-1b9b-42b5-9662-82db6865f2a4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xlh7s" Sep 14 11:08:42.491276 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:42.491234 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xlh7s" Sep 14 11:08:42.510257 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:42.510238 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5btpf\" (UniqueName: \"kubernetes.io/projected/c630b5a2-2f5a-4d78-afac-eeadc2062d1e-kube-api-access-5btpf\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:08:42.510336 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:42.510260 2556 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c630b5a2-2f5a-4d78-afac-eeadc2062d1e-extensions-socket-volume\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:08:42.772033 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:42.771942 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xlh7s"] Sep 14 11:08:42.776473 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:08:42.776444 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab728de_1b9b_42b5_9662_82db6865f2a4.slice/crio-71d9c178dd1ff7894fa364e7364b36f8383dcac8edf0255b503d76da7bef2dd6 WatchSource:0}: Error finding container 71d9c178dd1ff7894fa364e7364b36f8383dcac8edf0255b503d76da7bef2dd6: Status 404 returned error can't find the container with id 71d9c178dd1ff7894fa364e7364b36f8383dcac8edf0255b503d76da7bef2dd6 Sep 14 11:08:43.152529 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:43.152494 2556 generic.go:356] "Generic (PLEG): container finished" podID="c630b5a2-2f5a-4d78-afac-eeadc2062d1e" containerID="66ed933d6fb141e1b11c6b4c638a745ada316d0999e2698816441c2a807abac3" exitCode=0 Sep 14 11:08:43.152935 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:43.152549 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6v8f" Sep 14 11:08:43.152935 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:43.152593 2556 scope.go:117] "RemoveContainer" containerID="66ed933d6fb141e1b11c6b4c638a745ada316d0999e2698816441c2a807abac3" Sep 14 11:08:43.154550 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:43.154524 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xlh7s" event={"ID":"dab728de-1b9b-42b5-9662-82db6865f2a4","Type":"ContainerStarted","Data":"66059507168b800f17313c973e5169451afd711e27eadf6974d0b877e8437d09"} Sep 14 11:08:43.154651 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:43.154593 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xlh7s" Sep 14 11:08:43.154651 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:43.154625 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xlh7s" event={"ID":"dab728de-1b9b-42b5-9662-82db6865f2a4","Type":"ContainerStarted","Data":"71d9c178dd1ff7894fa364e7364b36f8383dcac8edf0255b503d76da7bef2dd6"} Sep 14 11:08:43.161435 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:43.161413 2556 scope.go:117] "RemoveContainer" containerID="66ed933d6fb141e1b11c6b4c638a745ada316d0999e2698816441c2a807abac3" Sep 14 11:08:43.161718 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:08:43.161699 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66ed933d6fb141e1b11c6b4c638a745ada316d0999e2698816441c2a807abac3\": container with ID starting with 66ed933d6fb141e1b11c6b4c638a745ada316d0999e2698816441c2a807abac3 not found: ID does not exist" containerID="66ed933d6fb141e1b11c6b4c638a745ada316d0999e2698816441c2a807abac3" Sep 14 11:08:43.161803 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:43.161726 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ed933d6fb141e1b11c6b4c638a745ada316d0999e2698816441c2a807abac3"} err="failed to get container status \"66ed933d6fb141e1b11c6b4c638a745ada316d0999e2698816441c2a807abac3\": rpc error: code = NotFound desc = could not find container \"66ed933d6fb141e1b11c6b4c638a745ada316d0999e2698816441c2a807abac3\": container with ID starting with 66ed933d6fb141e1b11c6b4c638a745ada316d0999e2698816441c2a807abac3 not found: ID does not exist" Sep 14 11:08:43.181063 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:43.180994 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xlh7s" podStartSLOduration=1.180976309 podStartE2EDuration="1.180976309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:08:43.180303763 +0000 UTC m=+481.243706664" watchObservedRunningTime="2026-09-14 11:08:43.180976309 +0000 UTC m=+481.244379223" Sep 14 11:08:43.219893 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:43.219852 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65"] Sep 14 11:08:43.224171 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:43.224147 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65" Sep 14 11:08:43.237727 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:43.237701 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65"] Sep 14 11:08:43.317833 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:43.317792 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbbzx\" (UniqueName: \"kubernetes.io/projected/32e5fce6-40e2-4456-bc82-d258c4ccf5d1-kube-api-access-bbbzx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5bn65\" (UID: \"32e5fce6-40e2-4456-bc82-d258c4ccf5d1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65" Sep 14 11:08:43.318021 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:43.317923 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32e5fce6-40e2-4456-bc82-d258c4ccf5d1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5bn65\" (UID: \"32e5fce6-40e2-4456-bc82-d258c4ccf5d1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65" Sep 14 11:08:43.418722 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:43.418637 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32e5fce6-40e2-4456-bc82-d258c4ccf5d1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5bn65\" (UID: \"32e5fce6-40e2-4456-bc82-d258c4ccf5d1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65" Sep 14 11:08:43.418722 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:43.418699 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbbzx\" (UniqueName: \"kubernetes.io/projected/32e5fce6-40e2-4456-bc82-d258c4ccf5d1-kube-api-access-bbbzx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5bn65\" (UID: \"32e5fce6-40e2-4456-bc82-d258c4ccf5d1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65" Sep 14 11:08:43.419029 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:43.419011 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32e5fce6-40e2-4456-bc82-d258c4ccf5d1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5bn65\" (UID: \"32e5fce6-40e2-4456-bc82-d258c4ccf5d1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65" Sep 14 11:08:43.489260 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:43.489222 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbbzx\" (UniqueName: \"kubernetes.io/projected/32e5fce6-40e2-4456-bc82-d258c4ccf5d1-kube-api-access-bbbzx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5bn65\" (UID: \"32e5fce6-40e2-4456-bc82-d258c4ccf5d1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65" Sep 14 11:08:43.534010 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:43.533967 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65" Sep 14 11:08:43.822973 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:43.822887 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65"] Sep 14 11:08:43.826322 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:08:43.826295 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e5fce6_40e2_4456_bc82_d258c4ccf5d1.slice/crio-a4f57bb8b21ef29d58ddb61602ccbb122fe7e17eadce2a8a258d34ef06499402 WatchSource:0}: Error finding container a4f57bb8b21ef29d58ddb61602ccbb122fe7e17eadce2a8a258d34ef06499402: Status 404 returned error can't find the container with id a4f57bb8b21ef29d58ddb61602ccbb122fe7e17eadce2a8a258d34ef06499402 Sep 14 11:08:44.160047 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:44.160002 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65" event={"ID":"32e5fce6-40e2-4456-bc82-d258c4ccf5d1","Type":"ContainerStarted","Data":"097c92e82e2c091f9d4d774626a8d5455ce600c3c207d340342f0075419db0df"} Sep 14 11:08:44.160488 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:44.160078 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65" Sep 14 11:08:44.160488 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:44.160129 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65" event={"ID":"32e5fce6-40e2-4456-bc82-d258c4ccf5d1","Type":"ContainerStarted","Data":"a4f57bb8b21ef29d58ddb61602ccbb122fe7e17eadce2a8a258d34ef06499402"} Sep 14 11:08:44.180280 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:44.180229 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65" podStartSLOduration=1.180211347 podStartE2EDuration="1.180211347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:08:44.179177971 +0000 UTC m=+482.242580883" watchObservedRunningTime="2026-09-14 11:08:44.180211347 +0000 UTC m=+482.243614259" Sep 14 11:08:44.382197 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:44.382163 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c630b5a2-2f5a-4d78-afac-eeadc2062d1e" path="/var/lib/kubelet/pods/c630b5a2-2f5a-4d78-afac-eeadc2062d1e/volumes" Sep 14 11:08:54.163553 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:54.163523 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xlh7s" Sep 14 11:08:55.168456 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:55.168426 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65" Sep 14 11:08:55.249645 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:55.249616 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xlh7s"] Sep 14 11:08:55.249828 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:55.249804 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xlh7s" podUID="dab728de-1b9b-42b5-9662-82db6865f2a4" containerName="manager" containerID="cri-o://66059507168b800f17313c973e5169451afd711e27eadf6974d0b877e8437d09" gracePeriod=10 Sep 14 11:08:55.481098 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:55.481066 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xlh7s" Sep 14 11:08:55.613980 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:55.613945 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dab728de-1b9b-42b5-9662-82db6865f2a4-extensions-socket-volume\") pod \"dab728de-1b9b-42b5-9662-82db6865f2a4\" (UID: \"dab728de-1b9b-42b5-9662-82db6865f2a4\") " Sep 14 11:08:55.614139 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:55.613997 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4j5gv\" (UniqueName: \"kubernetes.io/projected/dab728de-1b9b-42b5-9662-82db6865f2a4-kube-api-access-4j5gv\") pod \"dab728de-1b9b-42b5-9662-82db6865f2a4\" (UID: \"dab728de-1b9b-42b5-9662-82db6865f2a4\") " Sep 14 11:08:55.614300 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:55.614279 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab728de-1b9b-42b5-9662-82db6865f2a4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "dab728de-1b9b-42b5-9662-82db6865f2a4" (UID: "dab728de-1b9b-42b5-9662-82db6865f2a4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:08:55.615812 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:55.615792 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab728de-1b9b-42b5-9662-82db6865f2a4-kube-api-access-4j5gv" (OuterVolumeSpecName: "kube-api-access-4j5gv") pod "dab728de-1b9b-42b5-9662-82db6865f2a4" (UID: "dab728de-1b9b-42b5-9662-82db6865f2a4"). InnerVolumeSpecName "kube-api-access-4j5gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:08:55.714731 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:55.714677 2556 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dab728de-1b9b-42b5-9662-82db6865f2a4-extensions-socket-volume\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:08:55.714731 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:55.714697 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j5gv\" (UniqueName: \"kubernetes.io/projected/dab728de-1b9b-42b5-9662-82db6865f2a4-kube-api-access-4j5gv\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:08:56.214851 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:56.214821 2556 generic.go:356] "Generic (PLEG): container finished" podID="dab728de-1b9b-42b5-9662-82db6865f2a4" containerID="66059507168b800f17313c973e5169451afd711e27eadf6974d0b877e8437d09" exitCode=0 Sep 14 11:08:56.215271 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:56.214861 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xlh7s" Sep 14 11:08:56.215271 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:56.214880 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xlh7s" event={"ID":"dab728de-1b9b-42b5-9662-82db6865f2a4","Type":"ContainerDied","Data":"66059507168b800f17313c973e5169451afd711e27eadf6974d0b877e8437d09"} Sep 14 11:08:56.215271 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:56.214917 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xlh7s" event={"ID":"dab728de-1b9b-42b5-9662-82db6865f2a4","Type":"ContainerDied","Data":"71d9c178dd1ff7894fa364e7364b36f8383dcac8edf0255b503d76da7bef2dd6"} Sep 14 11:08:56.215271 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:56.214938 2556 scope.go:117] "RemoveContainer" containerID="66059507168b800f17313c973e5169451afd711e27eadf6974d0b877e8437d09" Sep 14 11:08:56.223266 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:56.223249 2556 scope.go:117] "RemoveContainer" containerID="66059507168b800f17313c973e5169451afd711e27eadf6974d0b877e8437d09" Sep 14 11:08:56.223503 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:08:56.223480 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66059507168b800f17313c973e5169451afd711e27eadf6974d0b877e8437d09\": container with ID starting with 66059507168b800f17313c973e5169451afd711e27eadf6974d0b877e8437d09 not found: ID does not exist" containerID="66059507168b800f17313c973e5169451afd711e27eadf6974d0b877e8437d09" Sep 14 11:08:56.223595 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:56.223512 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66059507168b800f17313c973e5169451afd711e27eadf6974d0b877e8437d09"} err="failed to get container status \"66059507168b800f17313c973e5169451afd711e27eadf6974d0b877e8437d09\": rpc error: code = NotFound desc = could not find container \"66059507168b800f17313c973e5169451afd711e27eadf6974d0b877e8437d09\": container with ID starting with 66059507168b800f17313c973e5169451afd711e27eadf6974d0b877e8437d09 not found: ID does not exist" Sep 14 11:08:56.239401 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:56.239377 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xlh7s"] Sep 14 11:08:56.242886 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:56.242864 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xlh7s"] Sep 14 11:08:56.381889 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:08:56.381849 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dab728de-1b9b-42b5-9662-82db6865f2a4" path="/var/lib/kubelet/pods/dab728de-1b9b-42b5-9662-82db6865f2a4/volumes" Sep 14 11:09:11.630384 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.630355 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s"] Sep 14 11:09:11.630889 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.630639 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dab728de-1b9b-42b5-9662-82db6865f2a4" containerName="manager" Sep 14 11:09:11.630889 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.630651 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="dab728de-1b9b-42b5-9662-82db6865f2a4" containerName="manager" Sep 14 11:09:11.630889 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.630707 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="dab728de-1b9b-42b5-9662-82db6865f2a4" containerName="manager" Sep 14 11:09:11.632643 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.632625 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.635025 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.635005 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-z947q\"" Sep 14 11:09:11.649744 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.649723 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s"] Sep 14 11:09:11.719218 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.719193 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.719317 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.719225 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.719317 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.719248 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.719317 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.719267 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.719428 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.719330 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.719428 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.719359 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.719428 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.719379 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.719428 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.719395 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czxx5\" (UniqueName: \"kubernetes.io/projected/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-kube-api-access-czxx5\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.719567 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.719468 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.820346 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.820324 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.820446 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.820352 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czxx5\" (UniqueName: \"kubernetes.io/projected/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-kube-api-access-czxx5\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.820446 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.820384 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.820446 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.820405 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.820591 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.820540 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.820645 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.820593 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.820645 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.820627 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.820746 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.820676 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.820746 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.820703 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.820833 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.820761 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.820981 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.820959 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.821045 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.821011 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.821116 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.821040 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.821279 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.821255 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.822935 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.822911 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.822935 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.822928 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.834797 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.834768 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czxx5\" (UniqueName: \"kubernetes.io/projected/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-kube-api-access-czxx5\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.834892 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.834850 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afd8f26-9ef1-47db-ac05-349b80a4c6b1-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s\" (UID: \"3afd8f26-9ef1-47db-ac05-349b80a4c6b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:11.941935 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:11.941869 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:12.317688 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:09:12.317660 2556 log.go:32] "RunPodSandbox from runtime service failed" err=< Sep 14 11:09:12.317688 ip-10-0-143-52 kubenswrapper[2556]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-78bb78ff78-fdw2s_openshift-ingress_3afd8f26-9ef1-47db-ac05-349b80a4c6b1_0(0b1dbb1d3e85c66f38c64e9f6276e5ffd418b707d6f85b4e23e4c12a6090153f): error adding pod openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fdw2s to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"0b1dbb1d3e85c66f38c64e9f6276e5ffd418b707d6f85b4e23e4c12a6090153f" Netns:"/var/run/netns/e1bbdd71-d8d5-412f-ae4f-3bbd6a5b8bce" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-78bb78ff78-fdw2s;K8S_POD_INFRA_CONTAINER_ID=0b1dbb1d3e85c66f38c64e9f6276e5ffd418b707d6f85b4e23e4c12a6090153f;K8S_POD_UID=3afd8f26-9ef1-47db-ac05-349b80a4c6b1" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s/3afd8f26-9ef1-47db-ac05-349b80a4c6b1]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod maas-default-gateway-openshift-default-78bb78ff78-fdw2s in out of cluster comm: SetNetworkStatus: failed to update the pod maas-default-gateway-openshift-default-78bb78ff78-fdw2s in out of cluster comm: status update failed for pod openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s: Internal error occurred: failed calling webhook "pod.network-node-identity.openshift.io": failed to call webhook: Post "https://network-node-identity.clusters-111b145b-0af2-4f05-b195-55b6cc4b454a.svc:9743/pod?timeout=10s": dial tcp 172.30.237.136:9743: connect: connection refused Sep 14 11:09:12.317688 ip-10-0-143-52 kubenswrapper[2556]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 14 11:09:12.317688 ip-10-0-143-52 kubenswrapper[2556]: > Sep 14 11:09:12.317841 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:09:12.317715 2556 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Sep 14 11:09:12.317841 ip-10-0-143-52 kubenswrapper[2556]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-78bb78ff78-fdw2s_openshift-ingress_3afd8f26-9ef1-47db-ac05-349b80a4c6b1_0(0b1dbb1d3e85c66f38c64e9f6276e5ffd418b707d6f85b4e23e4c12a6090153f): error adding pod openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fdw2s to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"0b1dbb1d3e85c66f38c64e9f6276e5ffd418b707d6f85b4e23e4c12a6090153f" Netns:"/var/run/netns/e1bbdd71-d8d5-412f-ae4f-3bbd6a5b8bce" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-78bb78ff78-fdw2s;K8S_POD_INFRA_CONTAINER_ID=0b1dbb1d3e85c66f38c64e9f6276e5ffd418b707d6f85b4e23e4c12a6090153f;K8S_POD_UID=3afd8f26-9ef1-47db-ac05-349b80a4c6b1" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s/3afd8f26-9ef1-47db-ac05-349b80a4c6b1]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod maas-default-gateway-openshift-default-78bb78ff78-fdw2s in out of cluster comm: SetNetworkStatus: failed to update the pod maas-default-gateway-openshift-default-78bb78ff78-fdw2s in out of cluster comm: status update failed for pod openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s: Internal error occurred: failed calling webhook "pod.network-node-identity.openshift.io": failed to call webhook: Post "https://network-node-identity.clusters-111b145b-0af2-4f05-b195-55b6cc4b454a.svc:9743/pod?timeout=10s": dial tcp 172.30.237.136:9743: connect: connection refused Sep 14 11:09:12.317841 ip-10-0-143-52 kubenswrapper[2556]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 14 11:09:12.317841 ip-10-0-143-52 kubenswrapper[2556]: > pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:12.317841 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:09:12.317734 2556 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=< Sep 14 11:09:12.317841 ip-10-0-143-52 kubenswrapper[2556]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-78bb78ff78-fdw2s_openshift-ingress_3afd8f26-9ef1-47db-ac05-349b80a4c6b1_0(0b1dbb1d3e85c66f38c64e9f6276e5ffd418b707d6f85b4e23e4c12a6090153f): error adding pod openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fdw2s to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"0b1dbb1d3e85c66f38c64e9f6276e5ffd418b707d6f85b4e23e4c12a6090153f" Netns:"/var/run/netns/e1bbdd71-d8d5-412f-ae4f-3bbd6a5b8bce" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-78bb78ff78-fdw2s;K8S_POD_INFRA_CONTAINER_ID=0b1dbb1d3e85c66f38c64e9f6276e5ffd418b707d6f85b4e23e4c12a6090153f;K8S_POD_UID=3afd8f26-9ef1-47db-ac05-349b80a4c6b1" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s/3afd8f26-9ef1-47db-ac05-349b80a4c6b1]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod maas-default-gateway-openshift-default-78bb78ff78-fdw2s in out of cluster comm: SetNetworkStatus: failed to update the pod maas-default-gateway-openshift-default-78bb78ff78-fdw2s in out of cluster comm: status update failed for pod openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s: Internal error occurred: failed calling webhook "pod.network-node-identity.openshift.io": failed to call webhook: Post "https://network-node-identity.clusters-111b145b-0af2-4f05-b195-55b6cc4b454a.svc:9743/pod?timeout=10s": dial tcp 172.30.237.136:9743: connect: connection refused Sep 14 11:09:12.317841 ip-10-0-143-52 kubenswrapper[2556]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 14 11:09:12.317841 ip-10-0-143-52 kubenswrapper[2556]: > pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:12.318113 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:09:12.317806 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s_openshift-ingress(3afd8f26-9ef1-47db-ac05-349b80a4c6b1)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-default-gateway-openshift-default-78bb78ff78-fdw2s_openshift-ingress(3afd8f26-9ef1-47db-ac05-349b80a4c6b1)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-78bb78ff78-fdw2s_openshift-ingress_3afd8f26-9ef1-47db-ac05-349b80a4c6b1_0(0b1dbb1d3e85c66f38c64e9f6276e5ffd418b707d6f85b4e23e4c12a6090153f): error adding pod openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fdw2s to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"0b1dbb1d3e85c66f38c64e9f6276e5ffd418b707d6f85b4e23e4c12a6090153f\\\" Netns:\\\"/var/run/netns/e1bbdd71-d8d5-412f-ae4f-3bbd6a5b8bce\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-78bb78ff78-fdw2s;K8S_POD_INFRA_CONTAINER_ID=0b1dbb1d3e85c66f38c64e9f6276e5ffd418b707d6f85b4e23e4c12a6090153f;K8S_POD_UID=3afd8f26-9ef1-47db-ac05-349b80a4c6b1\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s/3afd8f26-9ef1-47db-ac05-349b80a4c6b1]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod maas-default-gateway-openshift-default-78bb78ff78-fdw2s in out of cluster comm: SetNetworkStatus: failed to update the pod maas-default-gateway-openshift-default-78bb78ff78-fdw2s in out of cluster comm: status update failed for pod openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s: Internal error occurred: failed calling webhook \\\"pod.network-node-identity.openshift.io\\\": failed to call webhook: Post \\\"https://network-node-identity.clusters-111b145b-0af2-4f05-b195-55b6cc4b454a.svc:9743/pod?timeout=10s\\\": dial tcp 172.30.237.136:9743: connect: connection refused\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" podUID="3afd8f26-9ef1-47db-ac05-349b80a4c6b1" Sep 14 11:09:13.286875 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:13.286845 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:13.287266 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:13.287148 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:14.745593 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:09:14.745560 2556 log.go:32] "RunPodSandbox from runtime service failed" err=< Sep 14 11:09:14.745593 ip-10-0-143-52 kubenswrapper[2556]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-78bb78ff78-fdw2s_openshift-ingress_3afd8f26-9ef1-47db-ac05-349b80a4c6b1_0(5b9202ae9752e257bcc3ce65d9cb9998674877fea78e93e33614f0dcd947cdba): error adding pod openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fdw2s to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"5b9202ae9752e257bcc3ce65d9cb9998674877fea78e93e33614f0dcd947cdba" Netns:"/var/run/netns/336c36fb-cd81-4ebc-8ec6-f651135dec81" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-78bb78ff78-fdw2s;K8S_POD_INFRA_CONTAINER_ID=5b9202ae9752e257bcc3ce65d9cb9998674877fea78e93e33614f0dcd947cdba;K8S_POD_UID=3afd8f26-9ef1-47db-ac05-349b80a4c6b1" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s/3afd8f26-9ef1-47db-ac05-349b80a4c6b1]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod maas-default-gateway-openshift-default-78bb78ff78-fdw2s in out of cluster comm: SetNetworkStatus: failed to update the pod maas-default-gateway-openshift-default-78bb78ff78-fdw2s in out of cluster comm: status update failed for pod openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s: Internal error occurred: failed calling webhook "pod.network-node-identity.openshift.io": failed to call webhook: Post "https://network-node-identity.clusters-111b145b-0af2-4f05-b195-55b6cc4b454a.svc:9743/pod?timeout=10s": dial tcp 172.30.237.136:9743: connect: connection refused Sep 14 11:09:14.745593 ip-10-0-143-52 kubenswrapper[2556]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 14 11:09:14.745593 ip-10-0-143-52 kubenswrapper[2556]: > Sep 14 11:09:14.745954 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:09:14.745626 2556 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Sep 14 11:09:14.745954 ip-10-0-143-52 kubenswrapper[2556]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-78bb78ff78-fdw2s_openshift-ingress_3afd8f26-9ef1-47db-ac05-349b80a4c6b1_0(5b9202ae9752e257bcc3ce65d9cb9998674877fea78e93e33614f0dcd947cdba): error adding pod openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fdw2s to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"5b9202ae9752e257bcc3ce65d9cb9998674877fea78e93e33614f0dcd947cdba" Netns:"/var/run/netns/336c36fb-cd81-4ebc-8ec6-f651135dec81" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-78bb78ff78-fdw2s;K8S_POD_INFRA_CONTAINER_ID=5b9202ae9752e257bcc3ce65d9cb9998674877fea78e93e33614f0dcd947cdba;K8S_POD_UID=3afd8f26-9ef1-47db-ac05-349b80a4c6b1" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s/3afd8f26-9ef1-47db-ac05-349b80a4c6b1]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod maas-default-gateway-openshift-default-78bb78ff78-fdw2s in out of cluster comm: SetNetworkStatus: failed to update the pod maas-default-gateway-openshift-default-78bb78ff78-fdw2s in out of cluster comm: status update failed for pod openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s: Internal error occurred: failed calling webhook "pod.network-node-identity.openshift.io": failed to call webhook: Post "https://network-node-identity.clusters-111b145b-0af2-4f05-b195-55b6cc4b454a.svc:9743/pod?timeout=10s": dial tcp 172.30.237.136:9743: connect: connection refused Sep 14 11:09:14.745954 ip-10-0-143-52 kubenswrapper[2556]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 14 11:09:14.745954 ip-10-0-143-52 kubenswrapper[2556]: > pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:14.745954 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:09:14.745661 2556 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=< Sep 14 11:09:14.745954 ip-10-0-143-52 kubenswrapper[2556]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-78bb78ff78-fdw2s_openshift-ingress_3afd8f26-9ef1-47db-ac05-349b80a4c6b1_0(5b9202ae9752e257bcc3ce65d9cb9998674877fea78e93e33614f0dcd947cdba): error adding pod openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fdw2s to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"5b9202ae9752e257bcc3ce65d9cb9998674877fea78e93e33614f0dcd947cdba" Netns:"/var/run/netns/336c36fb-cd81-4ebc-8ec6-f651135dec81" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-78bb78ff78-fdw2s;K8S_POD_INFRA_CONTAINER_ID=5b9202ae9752e257bcc3ce65d9cb9998674877fea78e93e33614f0dcd947cdba;K8S_POD_UID=3afd8f26-9ef1-47db-ac05-349b80a4c6b1" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s/3afd8f26-9ef1-47db-ac05-349b80a4c6b1]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod maas-default-gateway-openshift-default-78bb78ff78-fdw2s in out of cluster comm: SetNetworkStatus: failed to update the pod maas-default-gateway-openshift-default-78bb78ff78-fdw2s in out of cluster comm: status update failed for pod openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s: Internal error occurred: failed calling webhook "pod.network-node-identity.openshift.io": failed to call webhook: Post "https://network-node-identity.clusters-111b145b-0af2-4f05-b195-55b6cc4b454a.svc:9743/pod?timeout=10s": dial tcp 172.30.237.136:9743: connect: connection refused Sep 14 11:09:14.745954 ip-10-0-143-52 kubenswrapper[2556]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 14 11:09:14.745954 ip-10-0-143-52 kubenswrapper[2556]: > pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:14.746210 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:09:14.745731 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s_openshift-ingress(3afd8f26-9ef1-47db-ac05-349b80a4c6b1)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-default-gateway-openshift-default-78bb78ff78-fdw2s_openshift-ingress(3afd8f26-9ef1-47db-ac05-349b80a4c6b1)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-78bb78ff78-fdw2s_openshift-ingress_3afd8f26-9ef1-47db-ac05-349b80a4c6b1_0(5b9202ae9752e257bcc3ce65d9cb9998674877fea78e93e33614f0dcd947cdba): error adding pod openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fdw2s to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"5b9202ae9752e257bcc3ce65d9cb9998674877fea78e93e33614f0dcd947cdba\\\" Netns:\\\"/var/run/netns/336c36fb-cd81-4ebc-8ec6-f651135dec81\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-78bb78ff78-fdw2s;K8S_POD_INFRA_CONTAINER_ID=5b9202ae9752e257bcc3ce65d9cb9998674877fea78e93e33614f0dcd947cdba;K8S_POD_UID=3afd8f26-9ef1-47db-ac05-349b80a4c6b1\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s/3afd8f26-9ef1-47db-ac05-349b80a4c6b1]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod maas-default-gateway-openshift-default-78bb78ff78-fdw2s in out of cluster comm: SetNetworkStatus: failed to update the pod maas-default-gateway-openshift-default-78bb78ff78-fdw2s in out of cluster comm: status update failed for pod openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s: Internal error occurred: failed calling webhook \\\"pod.network-node-identity.openshift.io\\\": failed to call webhook: Post \\\"https://network-node-identity.clusters-111b145b-0af2-4f05-b195-55b6cc4b454a.svc:9743/pod?timeout=10s\\\": dial tcp 172.30.237.136:9743: connect: connection refused\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" podUID="3afd8f26-9ef1-47db-ac05-349b80a4c6b1" Sep 14 11:09:27.376339 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:27.376308 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:27.376698 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:27.376556 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:27.779138 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:09:27.779065 2556 log.go:32] "RunPodSandbox from runtime service failed" err=< Sep 14 11:09:27.779138 ip-10-0-143-52 kubenswrapper[2556]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-78bb78ff78-fdw2s_openshift-ingress_3afd8f26-9ef1-47db-ac05-349b80a4c6b1_0(299d217a539c468636368ca3c8a5f86ab22a479dc4d97f2e2a24358fa666f0fb): error adding pod openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fdw2s to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"299d217a539c468636368ca3c8a5f86ab22a479dc4d97f2e2a24358fa666f0fb" Netns:"/var/run/netns/4a0790f4-925d-4d1f-b8d8-0237fdbf3116" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-78bb78ff78-fdw2s;K8S_POD_INFRA_CONTAINER_ID=299d217a539c468636368ca3c8a5f86ab22a479dc4d97f2e2a24358fa666f0fb;K8S_POD_UID=3afd8f26-9ef1-47db-ac05-349b80a4c6b1" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s/3afd8f26-9ef1-47db-ac05-349b80a4c6b1]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod maas-default-gateway-openshift-default-78bb78ff78-fdw2s in out of cluster comm: SetNetworkStatus: failed to update the pod maas-default-gateway-openshift-default-78bb78ff78-fdw2s in out of cluster comm: status update failed for pod openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s: Internal error occurred: failed calling webhook "pod.network-node-identity.openshift.io": failed to call webhook: Post "https://network-node-identity.clusters-111b145b-0af2-4f05-b195-55b6cc4b454a.svc:9743/pod?timeout=10s": dial tcp 172.30.237.136:9743: connect: connection refused Sep 14 11:09:27.779138 ip-10-0-143-52 kubenswrapper[2556]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 14 11:09:27.779138 ip-10-0-143-52 kubenswrapper[2556]: > Sep 14 11:09:27.779311 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:09:27.779164 2556 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Sep 14 11:09:27.779311 ip-10-0-143-52 kubenswrapper[2556]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-78bb78ff78-fdw2s_openshift-ingress_3afd8f26-9ef1-47db-ac05-349b80a4c6b1_0(299d217a539c468636368ca3c8a5f86ab22a479dc4d97f2e2a24358fa666f0fb): error adding pod openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fdw2s to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"299d217a539c468636368ca3c8a5f86ab22a479dc4d97f2e2a24358fa666f0fb" Netns:"/var/run/netns/4a0790f4-925d-4d1f-b8d8-0237fdbf3116" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-78bb78ff78-fdw2s;K8S_POD_INFRA_CONTAINER_ID=299d217a539c468636368ca3c8a5f86ab22a479dc4d97f2e2a24358fa666f0fb;K8S_POD_UID=3afd8f26-9ef1-47db-ac05-349b80a4c6b1" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s/3afd8f26-9ef1-47db-ac05-349b80a4c6b1]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod maas-default-gateway-openshift-default-78bb78ff78-fdw2s in out of cluster comm: SetNetworkStatus: failed to update the pod maas-default-gateway-openshift-default-78bb78ff78-fdw2s in out of cluster comm: status update failed for pod openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s: Internal error occurred: failed calling webhook "pod.network-node-identity.openshift.io": failed to call webhook: Post "https://network-node-identity.clusters-111b145b-0af2-4f05-b195-55b6cc4b454a.svc:9743/pod?timeout=10s": dial tcp 172.30.237.136:9743: connect: connection refused Sep 14 11:09:27.779311 ip-10-0-143-52 kubenswrapper[2556]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 14 11:09:27.779311 ip-10-0-143-52 kubenswrapper[2556]: > pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:27.779311 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:09:27.779182 2556 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=< Sep 14 11:09:27.779311 ip-10-0-143-52 kubenswrapper[2556]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-78bb78ff78-fdw2s_openshift-ingress_3afd8f26-9ef1-47db-ac05-349b80a4c6b1_0(299d217a539c468636368ca3c8a5f86ab22a479dc4d97f2e2a24358fa666f0fb): error adding pod openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fdw2s to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"299d217a539c468636368ca3c8a5f86ab22a479dc4d97f2e2a24358fa666f0fb" Netns:"/var/run/netns/4a0790f4-925d-4d1f-b8d8-0237fdbf3116" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-78bb78ff78-fdw2s;K8S_POD_INFRA_CONTAINER_ID=299d217a539c468636368ca3c8a5f86ab22a479dc4d97f2e2a24358fa666f0fb;K8S_POD_UID=3afd8f26-9ef1-47db-ac05-349b80a4c6b1" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s/3afd8f26-9ef1-47db-ac05-349b80a4c6b1]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod maas-default-gateway-openshift-default-78bb78ff78-fdw2s in out of cluster comm: SetNetworkStatus: failed to update the pod maas-default-gateway-openshift-default-78bb78ff78-fdw2s in out of cluster comm: status update failed for pod openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s: Internal error occurred: failed calling webhook "pod.network-node-identity.openshift.io": failed to call webhook: Post "https://network-node-identity.clusters-111b145b-0af2-4f05-b195-55b6cc4b454a.svc:9743/pod?timeout=10s": dial tcp 172.30.237.136:9743: connect: connection refused Sep 14 11:09:27.779311 ip-10-0-143-52 kubenswrapper[2556]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 14 11:09:27.779311 ip-10-0-143-52 kubenswrapper[2556]: > pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:27.779530 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:09:27.779266 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-default-gateway-openshift-default-78bb78ff78-fdw2s_openshift-ingress(3afd8f26-9ef1-47db-ac05-349b80a4c6b1)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-default-gateway-openshift-default-78bb78ff78-fdw2s_openshift-ingress(3afd8f26-9ef1-47db-ac05-349b80a4c6b1)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-78bb78ff78-fdw2s_openshift-ingress_3afd8f26-9ef1-47db-ac05-349b80a4c6b1_0(299d217a539c468636368ca3c8a5f86ab22a479dc4d97f2e2a24358fa666f0fb): error adding pod openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fdw2s to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"299d217a539c468636368ca3c8a5f86ab22a479dc4d97f2e2a24358fa666f0fb\\\" Netns:\\\"/var/run/netns/4a0790f4-925d-4d1f-b8d8-0237fdbf3116\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-78bb78ff78-fdw2s;K8S_POD_INFRA_CONTAINER_ID=299d217a539c468636368ca3c8a5f86ab22a479dc4d97f2e2a24358fa666f0fb;K8S_POD_UID=3afd8f26-9ef1-47db-ac05-349b80a4c6b1\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s/3afd8f26-9ef1-47db-ac05-349b80a4c6b1]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod maas-default-gateway-openshift-default-78bb78ff78-fdw2s in out of cluster comm: SetNetworkStatus: failed to update the pod maas-default-gateway-openshift-default-78bb78ff78-fdw2s in out of cluster comm: status update failed for pod openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s: Internal error occurred: failed calling webhook \\\"pod.network-node-identity.openshift.io\\\": failed to call webhook: Post \\\"https://network-node-identity.clusters-111b145b-0af2-4f05-b195-55b6cc4b454a.svc:9743/pod?timeout=10s\\\": dial tcp 172.30.237.136:9743: connect: connection refused\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" podUID="3afd8f26-9ef1-47db-ac05-349b80a4c6b1" Sep 14 11:09:40.376590 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:40.376514 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:40.376942 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:40.376840 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:40.714245 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:40.714160 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s"] Sep 14 11:09:40.718152 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:09:40.718121 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3afd8f26_9ef1_47db_ac05_349b80a4c6b1.slice/crio-3132da3d1d69fddc0ce618072713fc19e857a7e74186fa7ac1cd44d381745994 WatchSource:0}: Error finding container 3132da3d1d69fddc0ce618072713fc19e857a7e74186fa7ac1cd44d381745994: Status 404 returned error can't find the container with id 3132da3d1d69fddc0ce618072713fc19e857a7e74186fa7ac1cd44d381745994 Sep 14 11:09:40.720122 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:40.720074 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:09:40.720195 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:40.720175 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:09:40.720228 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:40.720211 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:09:41.407705 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:41.407664 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" event={"ID":"3afd8f26-9ef1-47db-ac05-349b80a4c6b1","Type":"ContainerStarted","Data":"8417d2fa10c6a652b7194b39c3e7d53a313740a7c3ce1b5dcf2800cdb9262d36"} Sep 14 11:09:41.408105 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:41.407726 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" event={"ID":"3afd8f26-9ef1-47db-ac05-349b80a4c6b1","Type":"ContainerStarted","Data":"3132da3d1d69fddc0ce618072713fc19e857a7e74186fa7ac1cd44d381745994"} Sep 14 11:09:41.442544 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:41.442498 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" podStartSLOduration=30.442484281 podStartE2EDuration="30.442484281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:09:41.442320623 +0000 UTC m=+539.505723533" watchObservedRunningTime="2026-09-14 11:09:41.442484281 +0000 UTC m=+539.505887191" Sep 14 11:09:41.942540 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:41.942512 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:41.942690 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:41.942563 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:41.946809 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:41.946787 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:42.412324 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:42.412286 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:09:47.778231 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:47.778197 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-lwh65"] Sep 14 11:09:47.782699 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:47.782682 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" Sep 14 11:09:47.785626 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:47.785607 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-sysctl-allowlist\"" Sep 14 11:09:47.862958 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:47.862922 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/80ee3da5-767b-4c8c-a93e-38021cbbda56-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-lwh65\" (UID: \"80ee3da5-767b-4c8c-a93e-38021cbbda56\") " pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" Sep 14 11:09:47.863131 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:47.863009 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80ee3da5-767b-4c8c-a93e-38021cbbda56-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-lwh65\" (UID: \"80ee3da5-767b-4c8c-a93e-38021cbbda56\") " pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" Sep 14 11:09:47.863131 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:47.863035 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4khkp\" (UniqueName: \"kubernetes.io/projected/80ee3da5-767b-4c8c-a93e-38021cbbda56-kube-api-access-4khkp\") pod \"cni-sysctl-allowlist-ds-lwh65\" (UID: \"80ee3da5-767b-4c8c-a93e-38021cbbda56\") " pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" Sep 14 11:09:47.863131 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:47.863051 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/80ee3da5-767b-4c8c-a93e-38021cbbda56-ready\") pod \"cni-sysctl-allowlist-ds-lwh65\" (UID: \"80ee3da5-767b-4c8c-a93e-38021cbbda56\") " pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" Sep 14 11:09:47.963803 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:47.963772 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80ee3da5-767b-4c8c-a93e-38021cbbda56-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-lwh65\" (UID: \"80ee3da5-767b-4c8c-a93e-38021cbbda56\") " pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" Sep 14 11:09:47.963803 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:47.963804 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4khkp\" (UniqueName: \"kubernetes.io/projected/80ee3da5-767b-4c8c-a93e-38021cbbda56-kube-api-access-4khkp\") pod \"cni-sysctl-allowlist-ds-lwh65\" (UID: \"80ee3da5-767b-4c8c-a93e-38021cbbda56\") " pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" Sep 14 11:09:47.963989 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:47.963821 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/80ee3da5-767b-4c8c-a93e-38021cbbda56-ready\") pod \"cni-sysctl-allowlist-ds-lwh65\" (UID: \"80ee3da5-767b-4c8c-a93e-38021cbbda56\") " pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" Sep 14 11:09:47.963989 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:47.963847 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/80ee3da5-767b-4c8c-a93e-38021cbbda56-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-lwh65\" (UID: \"80ee3da5-767b-4c8c-a93e-38021cbbda56\") " pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" Sep 14 11:09:47.963989 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:47.963917 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80ee3da5-767b-4c8c-a93e-38021cbbda56-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-lwh65\" (UID: \"80ee3da5-767b-4c8c-a93e-38021cbbda56\") " pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" Sep 14 11:09:47.964206 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:47.964191 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/80ee3da5-767b-4c8c-a93e-38021cbbda56-ready\") pod \"cni-sysctl-allowlist-ds-lwh65\" (UID: \"80ee3da5-767b-4c8c-a93e-38021cbbda56\") " pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" Sep 14 11:09:47.964429 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:47.964414 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/80ee3da5-767b-4c8c-a93e-38021cbbda56-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-lwh65\" (UID: \"80ee3da5-767b-4c8c-a93e-38021cbbda56\") " pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" Sep 14 11:09:47.976292 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:47.976268 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4khkp\" (UniqueName: \"kubernetes.io/projected/80ee3da5-767b-4c8c-a93e-38021cbbda56-kube-api-access-4khkp\") pod \"cni-sysctl-allowlist-ds-lwh65\" (UID: \"80ee3da5-767b-4c8c-a93e-38021cbbda56\") " pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" Sep 14 11:09:48.091587 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:48.091497 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" Sep 14 11:09:48.103423 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:09:48.103392 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ee3da5_767b_4c8c_a93e_38021cbbda56.slice/crio-acd1488cbfa23d332ab757fcb3367d0201f1e13328f4f3aaddc7382a6cfd90e9 WatchSource:0}: Error finding container acd1488cbfa23d332ab757fcb3367d0201f1e13328f4f3aaddc7382a6cfd90e9: Status 404 returned error can't find the container with id acd1488cbfa23d332ab757fcb3367d0201f1e13328f4f3aaddc7382a6cfd90e9 Sep 14 11:09:48.436005 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:48.435967 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" event={"ID":"80ee3da5-767b-4c8c-a93e-38021cbbda56","Type":"ContainerStarted","Data":"00ec68902b035181ff7079853e9e977fc0d9b2fdd03040c185f90fe1da79595d"} Sep 14 11:09:48.436184 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:48.436017 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" event={"ID":"80ee3da5-767b-4c8c-a93e-38021cbbda56","Type":"ContainerStarted","Data":"acd1488cbfa23d332ab757fcb3367d0201f1e13328f4f3aaddc7382a6cfd90e9"} Sep 14 11:09:48.436184 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:48.436172 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" Sep 14 11:09:48.464540 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:48.464489 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" podStartSLOduration=1.464472996 podStartE2EDuration="1.464472996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:09:48.464105402 +0000 UTC m=+546.527508303" watchObservedRunningTime="2026-09-14 11:09:48.464472996 +0000 UTC m=+546.527875904" Sep 14 11:09:49.450003 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:49.449976 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" Sep 14 11:09:49.938336 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:49.938303 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mpdl6"] Sep 14 11:09:49.941497 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:49.941477 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mpdl6" Sep 14 11:09:49.946133 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:49.946114 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pd7ff\"" Sep 14 11:09:49.981195 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:49.981171 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjwls\" (UniqueName: \"kubernetes.io/projected/ee963889-560a-440a-9edc-271358e0677b-kube-api-access-bjwls\") pod \"authorino-7bcfd8f9cd-mpdl6\" (UID: \"ee963889-560a-440a-9edc-271358e0677b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-mpdl6" Sep 14 11:09:49.984441 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:49.984418 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mpdl6"] Sep 14 11:09:50.081916 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:50.081877 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjwls\" (UniqueName: \"kubernetes.io/projected/ee963889-560a-440a-9edc-271358e0677b-kube-api-access-bjwls\") pod \"authorino-7bcfd8f9cd-mpdl6\" (UID: \"ee963889-560a-440a-9edc-271358e0677b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-mpdl6" Sep 14 11:09:50.090493 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:50.090463 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjwls\" (UniqueName: \"kubernetes.io/projected/ee963889-560a-440a-9edc-271358e0677b-kube-api-access-bjwls\") pod \"authorino-7bcfd8f9cd-mpdl6\" (UID: \"ee963889-560a-440a-9edc-271358e0677b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-mpdl6" Sep 14 11:09:50.171949 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:50.171912 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-nsbvd"] Sep 14 11:09:50.175100 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:50.175073 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-nsbvd" Sep 14 11:09:50.188197 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:50.188170 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-nsbvd"] Sep 14 11:09:50.250239 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:50.250135 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mpdl6" Sep 14 11:09:50.284328 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:50.284296 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q924\" (UniqueName: \"kubernetes.io/projected/b434ee77-d506-4948-8c88-56526fa1a419-kube-api-access-7q924\") pod \"authorino-59c974bd48-nsbvd\" (UID: \"b434ee77-d506-4948-8c88-56526fa1a419\") " pod="kuadrant-system/authorino-59c974bd48-nsbvd" Sep 14 11:09:50.384702 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:50.384669 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q924\" (UniqueName: \"kubernetes.io/projected/b434ee77-d506-4948-8c88-56526fa1a419-kube-api-access-7q924\") pod \"authorino-59c974bd48-nsbvd\" (UID: \"b434ee77-d506-4948-8c88-56526fa1a419\") " pod="kuadrant-system/authorino-59c974bd48-nsbvd" Sep 14 11:09:50.395720 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:50.395689 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q924\" (UniqueName: \"kubernetes.io/projected/b434ee77-d506-4948-8c88-56526fa1a419-kube-api-access-7q924\") pod \"authorino-59c974bd48-nsbvd\" (UID: \"b434ee77-d506-4948-8c88-56526fa1a419\") " pod="kuadrant-system/authorino-59c974bd48-nsbvd" Sep 14 11:09:50.451595 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:50.451538 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-lwh65"] Sep 14 11:09:50.484400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:50.484364 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-nsbvd" Sep 14 11:09:50.523962 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:50.523819 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mpdl6"] Sep 14 11:09:50.526104 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:09:50.526067 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee963889_560a_440a_9edc_271358e0677b.slice/crio-71786887a07653a10862be809c28e5d69f5f33061660375632d633b8ba4d87fc WatchSource:0}: Error finding container 71786887a07653a10862be809c28e5d69f5f33061660375632d633b8ba4d87fc: Status 404 returned error can't find the container with id 71786887a07653a10862be809c28e5d69f5f33061660375632d633b8ba4d87fc Sep 14 11:09:50.765339 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:50.765314 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-nsbvd"] Sep 14 11:09:50.767472 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:09:50.767443 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb434ee77_d506_4948_8c88_56526fa1a419.slice/crio-1672696eda46f10cbeb8b22aa0c15aad3a8a8c7217256b6e67239ae8f36bcd7b WatchSource:0}: Error finding container 1672696eda46f10cbeb8b22aa0c15aad3a8a8c7217256b6e67239ae8f36bcd7b: Status 404 returned error can't find the container with id 1672696eda46f10cbeb8b22aa0c15aad3a8a8c7217256b6e67239ae8f36bcd7b Sep 14 11:09:51.450599 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:51.450554 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mpdl6" event={"ID":"ee963889-560a-440a-9edc-271358e0677b","Type":"ContainerStarted","Data":"71786887a07653a10862be809c28e5d69f5f33061660375632d633b8ba4d87fc"} Sep 14 11:09:51.451634 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:51.451611 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-nsbvd" event={"ID":"b434ee77-d506-4948-8c88-56526fa1a419","Type":"ContainerStarted","Data":"1672696eda46f10cbeb8b22aa0c15aad3a8a8c7217256b6e67239ae8f36bcd7b"} Sep 14 11:09:51.451975 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:51.451720 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" podUID="80ee3da5-767b-4c8c-a93e-38021cbbda56" containerName="kube-multus-additional-cni-plugins" containerID="cri-o://00ec68902b035181ff7079853e9e977fc0d9b2fdd03040c185f90fe1da79595d" gracePeriod=30 Sep 14 11:09:54.464463 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:54.464428 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mpdl6" event={"ID":"ee963889-560a-440a-9edc-271358e0677b","Type":"ContainerStarted","Data":"ba824015d36abbbf6797ece5bb2f4557485d8cf86735cc6c5d5a047f0a2c1ee1"} Sep 14 11:09:54.465822 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:54.465795 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-nsbvd" event={"ID":"b434ee77-d506-4948-8c88-56526fa1a419","Type":"ContainerStarted","Data":"dc0dc050841c1ef034ef20f180c3b161844a89da7e4a5740bce59ff0281a0759"} Sep 14 11:09:54.483253 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:54.483210 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-mpdl6" podStartSLOduration=2.402901269 podStartE2EDuration="5.483197746s" totalImagesPullingTime="3.080296477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:09:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:09:54.482612027 +0000 UTC m=+552.546014937" watchObservedRunningTime="2026-09-14 11:09:54.483197746 +0000 UTC m=+552.546600655" Sep 14 11:09:54.541322 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:54.541273 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-nsbvd" podStartSLOduration=1.702886208 podStartE2EDuration="4.54125473s" totalImagesPullingTime="2.838368522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:09:54.504770854 +0000 UTC m=+552.568173764" watchObservedRunningTime="2026-09-14 11:09:54.54125473 +0000 UTC m=+552.604657639" Sep 14 11:09:54.542031 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:54.542013 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mpdl6"] Sep 14 11:09:56.473583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:56.473544 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-mpdl6" podUID="ee963889-560a-440a-9edc-271358e0677b" containerName="authorino" containerID="cri-o://ba824015d36abbbf6797ece5bb2f4557485d8cf86735cc6c5d5a047f0a2c1ee1" gracePeriod=30 Sep 14 11:09:56.712552 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:56.712527 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mpdl6" Sep 14 11:09:56.839987 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:56.839895 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjwls\" (UniqueName: \"kubernetes.io/projected/ee963889-560a-440a-9edc-271358e0677b-kube-api-access-bjwls\") pod \"ee963889-560a-440a-9edc-271358e0677b\" (UID: \"ee963889-560a-440a-9edc-271358e0677b\") " Sep 14 11:09:56.841877 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:56.841848 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee963889-560a-440a-9edc-271358e0677b-kube-api-access-bjwls" (OuterVolumeSpecName: "kube-api-access-bjwls") pod "ee963889-560a-440a-9edc-271358e0677b" (UID: "ee963889-560a-440a-9edc-271358e0677b"). InnerVolumeSpecName "kube-api-access-bjwls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:09:56.941425 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:56.941388 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjwls\" (UniqueName: \"kubernetes.io/projected/ee963889-560a-440a-9edc-271358e0677b-kube-api-access-bjwls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:09:57.478641 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:57.478610 2556 generic.go:356] "Generic (PLEG): container finished" podID="ee963889-560a-440a-9edc-271358e0677b" containerID="ba824015d36abbbf6797ece5bb2f4557485d8cf86735cc6c5d5a047f0a2c1ee1" exitCode=0 Sep 14 11:09:57.479075 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:57.478655 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mpdl6" Sep 14 11:09:57.479075 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:57.478670 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mpdl6" event={"ID":"ee963889-560a-440a-9edc-271358e0677b","Type":"ContainerDied","Data":"ba824015d36abbbf6797ece5bb2f4557485d8cf86735cc6c5d5a047f0a2c1ee1"} Sep 14 11:09:57.479075 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:57.478700 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mpdl6" event={"ID":"ee963889-560a-440a-9edc-271358e0677b","Type":"ContainerDied","Data":"71786887a07653a10862be809c28e5d69f5f33061660375632d633b8ba4d87fc"} Sep 14 11:09:57.479075 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:57.478716 2556 scope.go:117] "RemoveContainer" containerID="ba824015d36abbbf6797ece5bb2f4557485d8cf86735cc6c5d5a047f0a2c1ee1" Sep 14 11:09:57.486850 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:57.486831 2556 scope.go:117] "RemoveContainer" containerID="ba824015d36abbbf6797ece5bb2f4557485d8cf86735cc6c5d5a047f0a2c1ee1" Sep 14 11:09:57.487132 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:09:57.487105 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba824015d36abbbf6797ece5bb2f4557485d8cf86735cc6c5d5a047f0a2c1ee1\": container with ID starting with ba824015d36abbbf6797ece5bb2f4557485d8cf86735cc6c5d5a047f0a2c1ee1 not found: ID does not exist" containerID="ba824015d36abbbf6797ece5bb2f4557485d8cf86735cc6c5d5a047f0a2c1ee1" Sep 14 11:09:57.487193 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:57.487141 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba824015d36abbbf6797ece5bb2f4557485d8cf86735cc6c5d5a047f0a2c1ee1"} err="failed to get container status \"ba824015d36abbbf6797ece5bb2f4557485d8cf86735cc6c5d5a047f0a2c1ee1\": rpc error: code = NotFound desc = could not find container \"ba824015d36abbbf6797ece5bb2f4557485d8cf86735cc6c5d5a047f0a2c1ee1\": container with ID starting with ba824015d36abbbf6797ece5bb2f4557485d8cf86735cc6c5d5a047f0a2c1ee1 not found: ID does not exist" Sep 14 11:09:57.501145 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:57.501119 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mpdl6"] Sep 14 11:09:57.504744 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:57.504725 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mpdl6"] Sep 14 11:09:58.382104 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:09:58.382057 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee963889-560a-440a-9edc-271358e0677b" path="/var/lib/kubelet/pods/ee963889-560a-440a-9edc-271358e0677b/volumes" Sep 14 11:09:59.442685 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:09:59.442638 2556 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="00ec68902b035181ff7079853e9e977fc0d9b2fdd03040c185f90fe1da79595d" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 14 11:09:59.443623 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:09:59.443592 2556 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="00ec68902b035181ff7079853e9e977fc0d9b2fdd03040c185f90fe1da79595d" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 14 11:09:59.444431 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:09:59.444401 2556 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="00ec68902b035181ff7079853e9e977fc0d9b2fdd03040c185f90fe1da79595d" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 14 11:09:59.444513 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:09:59.444446 2556 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" podUID="80ee3da5-767b-4c8c-a93e-38021cbbda56" containerName="kube-multus-additional-cni-plugins" probeResult="unknown" Sep 14 11:10:09.442549 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:10:09.442508 2556 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="00ec68902b035181ff7079853e9e977fc0d9b2fdd03040c185f90fe1da79595d" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 14 11:10:09.443480 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:10:09.443455 2556 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="00ec68902b035181ff7079853e9e977fc0d9b2fdd03040c185f90fe1da79595d" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 14 11:10:09.444316 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:10:09.444296 2556 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="00ec68902b035181ff7079853e9e977fc0d9b2fdd03040c185f90fe1da79595d" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 14 11:10:09.444389 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:10:09.444325 2556 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" podUID="80ee3da5-767b-4c8c-a93e-38021cbbda56" containerName="kube-multus-additional-cni-plugins" probeResult="unknown" Sep 14 11:10:18.034821 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:18.034786 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-dl48j"] Sep 14 11:10:18.035231 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:18.035062 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee963889-560a-440a-9edc-271358e0677b" containerName="authorino" Sep 14 11:10:18.035231 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:18.035073 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee963889-560a-440a-9edc-271358e0677b" containerName="authorino" Sep 14 11:10:18.035231 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:18.035146 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee963889-560a-440a-9edc-271358e0677b" containerName="authorino" Sep 14 11:10:18.040587 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:18.040567 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dl48j" Sep 14 11:10:18.047391 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:18.047367 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-dl48j"] Sep 14 11:10:18.215004 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:18.214967 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5786\" (UniqueName: \"kubernetes.io/projected/c359d9a7-68fe-4b72-bb7e-fd32e2d86d8f-kube-api-access-j5786\") pod \"authorino-557f876f69-dl48j\" (UID: \"c359d9a7-68fe-4b72-bb7e-fd32e2d86d8f\") " pod="kuadrant-system/authorino-557f876f69-dl48j" Sep 14 11:10:18.296675 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:18.296596 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-dl48j"] Sep 14 11:10:18.296856 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:10:18.296836 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-j5786], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-dl48j" podUID="c359d9a7-68fe-4b72-bb7e-fd32e2d86d8f" Sep 14 11:10:18.316379 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:18.316344 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5786\" (UniqueName: \"kubernetes.io/projected/c359d9a7-68fe-4b72-bb7e-fd32e2d86d8f-kube-api-access-j5786\") pod \"authorino-557f876f69-dl48j\" (UID: \"c359d9a7-68fe-4b72-bb7e-fd32e2d86d8f\") " pod="kuadrant-system/authorino-557f876f69-dl48j" Sep 14 11:10:18.332448 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:18.332415 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5786\" (UniqueName: \"kubernetes.io/projected/c359d9a7-68fe-4b72-bb7e-fd32e2d86d8f-kube-api-access-j5786\") pod \"authorino-557f876f69-dl48j\" (UID: \"c359d9a7-68fe-4b72-bb7e-fd32e2d86d8f\") " pod="kuadrant-system/authorino-557f876f69-dl48j" Sep 14 11:10:18.343419 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:18.343394 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f97b8968-sbpck"] Sep 14 11:10:18.347406 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:18.347389 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f97b8968-sbpck" Sep 14 11:10:18.361928 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:18.361901 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f97b8968-sbpck"] Sep 14 11:10:18.518251 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:18.518203 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb7wq\" (UniqueName: \"kubernetes.io/projected/59dfb79b-6e56-4307-80cf-443e4ea81b75-kube-api-access-pb7wq\") pod \"authorino-5f97b8968-sbpck\" (UID: \"59dfb79b-6e56-4307-80cf-443e4ea81b75\") " pod="kuadrant-system/authorino-5f97b8968-sbpck" Sep 14 11:10:18.562942 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:18.562855 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dl48j" Sep 14 11:10:18.570190 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:18.570163 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dl48j" Sep 14 11:10:18.619257 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:18.619220 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb7wq\" (UniqueName: \"kubernetes.io/projected/59dfb79b-6e56-4307-80cf-443e4ea81b75-kube-api-access-pb7wq\") pod \"authorino-5f97b8968-sbpck\" (UID: \"59dfb79b-6e56-4307-80cf-443e4ea81b75\") " pod="kuadrant-system/authorino-5f97b8968-sbpck" Sep 14 11:10:18.633381 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:18.633345 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb7wq\" (UniqueName: \"kubernetes.io/projected/59dfb79b-6e56-4307-80cf-443e4ea81b75-kube-api-access-pb7wq\") pod \"authorino-5f97b8968-sbpck\" (UID: \"59dfb79b-6e56-4307-80cf-443e4ea81b75\") " pod="kuadrant-system/authorino-5f97b8968-sbpck" Sep 14 11:10:18.656365 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:18.656329 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f97b8968-sbpck" Sep 14 11:10:18.719741 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:18.719712 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5786\" (UniqueName: \"kubernetes.io/projected/c359d9a7-68fe-4b72-bb7e-fd32e2d86d8f-kube-api-access-j5786\") pod \"c359d9a7-68fe-4b72-bb7e-fd32e2d86d8f\" (UID: \"c359d9a7-68fe-4b72-bb7e-fd32e2d86d8f\") " Sep 14 11:10:18.722684 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:18.722655 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c359d9a7-68fe-4b72-bb7e-fd32e2d86d8f-kube-api-access-j5786" (OuterVolumeSpecName: "kube-api-access-j5786") pod "c359d9a7-68fe-4b72-bb7e-fd32e2d86d8f" (UID: "c359d9a7-68fe-4b72-bb7e-fd32e2d86d8f"). InnerVolumeSpecName "kube-api-access-j5786". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:10:18.820306 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:18.820229 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5786\" (UniqueName: \"kubernetes.io/projected/c359d9a7-68fe-4b72-bb7e-fd32e2d86d8f-kube-api-access-j5786\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:10:18.928783 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:18.928672 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f97b8968-sbpck"] Sep 14 11:10:18.930690 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:10:18.930666 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59dfb79b_6e56_4307_80cf_443e4ea81b75.slice/crio-49230a4a6aa3753877ba4857c2ab159afa09264a021baf66fe32785a97d7bb65 WatchSource:0}: Error finding container 49230a4a6aa3753877ba4857c2ab159afa09264a021baf66fe32785a97d7bb65: Status 404 returned error can't find the container with id 49230a4a6aa3753877ba4857c2ab159afa09264a021baf66fe32785a97d7bb65 Sep 14 11:10:19.442191 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:10:19.442140 2556 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="00ec68902b035181ff7079853e9e977fc0d9b2fdd03040c185f90fe1da79595d" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 14 11:10:19.443212 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:10:19.443180 2556 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="00ec68902b035181ff7079853e9e977fc0d9b2fdd03040c185f90fe1da79595d" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 14 11:10:19.444145 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:10:19.444121 2556 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="00ec68902b035181ff7079853e9e977fc0d9b2fdd03040c185f90fe1da79595d" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 14 11:10:19.444210 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:10:19.444159 2556 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" podUID="80ee3da5-767b-4c8c-a93e-38021cbbda56" containerName="kube-multus-additional-cni-plugins" probeResult="unknown" Sep 14 11:10:19.568188 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:19.568149 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f97b8968-sbpck" event={"ID":"59dfb79b-6e56-4307-80cf-443e4ea81b75","Type":"ContainerStarted","Data":"b02cc8cab9bd67b1918d25e37f25ae904f0540af212731ee0d60260fe864eef2"} Sep 14 11:10:19.568342 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:19.568196 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f97b8968-sbpck" event={"ID":"59dfb79b-6e56-4307-80cf-443e4ea81b75","Type":"ContainerStarted","Data":"49230a4a6aa3753877ba4857c2ab159afa09264a021baf66fe32785a97d7bb65"} Sep 14 11:10:19.568342 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:19.568239 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dl48j" Sep 14 11:10:19.589688 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:19.589637 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5f97b8968-sbpck" podStartSLOduration=1.159443203 podStartE2EDuration="1.589621654s" totalImagesPullingTime="430.178451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:10:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:10:19.58796857 +0000 UTC m=+577.651371477" watchObservedRunningTime="2026-09-14 11:10:19.589621654 +0000 UTC m=+577.653024563" Sep 14 11:10:19.630082 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:19.630006 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-dl48j"] Sep 14 11:10:19.636080 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:19.636052 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-nsbvd"] Sep 14 11:10:19.636294 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:19.636270 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-nsbvd" podUID="b434ee77-d506-4948-8c88-56526fa1a419" containerName="authorino" containerID="cri-o://dc0dc050841c1ef034ef20f180c3b161844a89da7e4a5740bce59ff0281a0759" gracePeriod=30 Sep 14 11:10:19.643916 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:19.643886 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-dl48j"] Sep 14 11:10:19.767425 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:19.767390 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5bd5697d46-8dr5t"] Sep 14 11:10:19.770502 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:19.770484 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bd5697d46-8dr5t" Sep 14 11:10:19.775224 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:19.775199 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 14 11:10:19.778233 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:19.778203 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5bd5697d46-8dr5t"] Sep 14 11:10:19.877241 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:19.877217 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-nsbvd" Sep 14 11:10:19.930070 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:19.929998 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e-tls-cert\") pod \"authorino-5bd5697d46-8dr5t\" (UID: \"92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e\") " pod="kuadrant-system/authorino-5bd5697d46-8dr5t" Sep 14 11:10:19.930070 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:19.930063 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bm52\" (UniqueName: \"kubernetes.io/projected/92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e-kube-api-access-9bm52\") pod \"authorino-5bd5697d46-8dr5t\" (UID: \"92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e\") " pod="kuadrant-system/authorino-5bd5697d46-8dr5t" Sep 14 11:10:20.030429 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:20.030393 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7q924\" (UniqueName: \"kubernetes.io/projected/b434ee77-d506-4948-8c88-56526fa1a419-kube-api-access-7q924\") pod \"b434ee77-d506-4948-8c88-56526fa1a419\" (UID: \"b434ee77-d506-4948-8c88-56526fa1a419\") " Sep 14 11:10:20.030573 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:20.030498 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bm52\" (UniqueName: \"kubernetes.io/projected/92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e-kube-api-access-9bm52\") pod \"authorino-5bd5697d46-8dr5t\" (UID: \"92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e\") " pod="kuadrant-system/authorino-5bd5697d46-8dr5t" Sep 14 11:10:20.030573 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:20.030535 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e-tls-cert\") pod \"authorino-5bd5697d46-8dr5t\" (UID: \"92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e\") " pod="kuadrant-system/authorino-5bd5697d46-8dr5t" Sep 14 11:10:20.032432 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:20.032399 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b434ee77-d506-4948-8c88-56526fa1a419-kube-api-access-7q924" (OuterVolumeSpecName: "kube-api-access-7q924") pod "b434ee77-d506-4948-8c88-56526fa1a419" (UID: "b434ee77-d506-4948-8c88-56526fa1a419"). InnerVolumeSpecName "kube-api-access-7q924". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:10:20.032870 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:20.032852 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e-tls-cert\") pod \"authorino-5bd5697d46-8dr5t\" (UID: \"92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e\") " pod="kuadrant-system/authorino-5bd5697d46-8dr5t" Sep 14 11:10:20.039223 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:20.039200 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bm52\" (UniqueName: \"kubernetes.io/projected/92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e-kube-api-access-9bm52\") pod \"authorino-5bd5697d46-8dr5t\" (UID: \"92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e\") " pod="kuadrant-system/authorino-5bd5697d46-8dr5t" Sep 14 11:10:20.080201 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:20.080169 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bd5697d46-8dr5t" Sep 14 11:10:20.131814 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:20.131783 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q924\" (UniqueName: \"kubernetes.io/projected/b434ee77-d506-4948-8c88-56526fa1a419-kube-api-access-7q924\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:10:20.371395 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:20.371373 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5bd5697d46-8dr5t"] Sep 14 11:10:20.372671 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:10:20.372644 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92ae7246_1fa8_4ee4_aae3_6b51b2f00d0e.slice/crio-dde403831314c17a5a94f30c5bcf807ab28542b891caea598263c102dc1d6ee4 WatchSource:0}: Error finding container dde403831314c17a5a94f30c5bcf807ab28542b891caea598263c102dc1d6ee4: Status 404 returned error can't find the container with id dde403831314c17a5a94f30c5bcf807ab28542b891caea598263c102dc1d6ee4 Sep 14 11:10:20.382235 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:20.382213 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c359d9a7-68fe-4b72-bb7e-fd32e2d86d8f" path="/var/lib/kubelet/pods/c359d9a7-68fe-4b72-bb7e-fd32e2d86d8f/volumes" Sep 14 11:10:20.573309 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:20.573277 2556 generic.go:356] "Generic (PLEG): container finished" podID="b434ee77-d506-4948-8c88-56526fa1a419" containerID="dc0dc050841c1ef034ef20f180c3b161844a89da7e4a5740bce59ff0281a0759" exitCode=0 Sep 14 11:10:20.573691 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:20.573321 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-nsbvd" Sep 14 11:10:20.573691 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:20.573354 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-nsbvd" event={"ID":"b434ee77-d506-4948-8c88-56526fa1a419","Type":"ContainerDied","Data":"dc0dc050841c1ef034ef20f180c3b161844a89da7e4a5740bce59ff0281a0759"} Sep 14 11:10:20.573691 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:20.573394 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-nsbvd" event={"ID":"b434ee77-d506-4948-8c88-56526fa1a419","Type":"ContainerDied","Data":"1672696eda46f10cbeb8b22aa0c15aad3a8a8c7217256b6e67239ae8f36bcd7b"} Sep 14 11:10:20.573691 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:20.573419 2556 scope.go:117] "RemoveContainer" containerID="dc0dc050841c1ef034ef20f180c3b161844a89da7e4a5740bce59ff0281a0759" Sep 14 11:10:20.574723 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:20.574705 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bd5697d46-8dr5t" event={"ID":"92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e","Type":"ContainerStarted","Data":"dde403831314c17a5a94f30c5bcf807ab28542b891caea598263c102dc1d6ee4"} Sep 14 11:10:20.580679 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:20.580664 2556 scope.go:117] "RemoveContainer" containerID="dc0dc050841c1ef034ef20f180c3b161844a89da7e4a5740bce59ff0281a0759" Sep 14 11:10:20.580926 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:10:20.580909 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc0dc050841c1ef034ef20f180c3b161844a89da7e4a5740bce59ff0281a0759\": container with ID starting with dc0dc050841c1ef034ef20f180c3b161844a89da7e4a5740bce59ff0281a0759 not found: ID does not exist" containerID="dc0dc050841c1ef034ef20f180c3b161844a89da7e4a5740bce59ff0281a0759" Sep 14 11:10:20.580987 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:20.580936 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc0dc050841c1ef034ef20f180c3b161844a89da7e4a5740bce59ff0281a0759"} err="failed to get container status \"dc0dc050841c1ef034ef20f180c3b161844a89da7e4a5740bce59ff0281a0759\": rpc error: code = NotFound desc = could not find container \"dc0dc050841c1ef034ef20f180c3b161844a89da7e4a5740bce59ff0281a0759\": container with ID starting with dc0dc050841c1ef034ef20f180c3b161844a89da7e4a5740bce59ff0281a0759 not found: ID does not exist" Sep 14 11:10:20.597125 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:20.597082 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-nsbvd"] Sep 14 11:10:20.606980 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:20.606952 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-nsbvd"] Sep 14 11:10:21.532452 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.532428 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-lwh65_80ee3da5-767b-4c8c-a93e-38021cbbda56/kube-multus-additional-cni-plugins/0.log" Sep 14 11:10:21.534573 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.534558 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" Sep 14 11:10:21.584535 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.584506 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bd5697d46-8dr5t" event={"ID":"92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e","Type":"ContainerStarted","Data":"289fbc516258695866ad0a5f723635ffb37d9ad441654618bb1edcf9ad8e78a4"} Sep 14 11:10:21.585689 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.585671 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-lwh65_80ee3da5-767b-4c8c-a93e-38021cbbda56/kube-multus-additional-cni-plugins/0.log" Sep 14 11:10:21.585781 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.585704 2556 generic.go:356] "Generic (PLEG): container finished" podID="80ee3da5-767b-4c8c-a93e-38021cbbda56" containerID="00ec68902b035181ff7079853e9e977fc0d9b2fdd03040c185f90fe1da79595d" exitCode=137 Sep 14 11:10:21.585781 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.585737 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" Sep 14 11:10:21.585859 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.585784 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" event={"ID":"80ee3da5-767b-4c8c-a93e-38021cbbda56","Type":"ContainerDied","Data":"00ec68902b035181ff7079853e9e977fc0d9b2fdd03040c185f90fe1da79595d"} Sep 14 11:10:21.585859 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.585825 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-lwh65" event={"ID":"80ee3da5-767b-4c8c-a93e-38021cbbda56","Type":"ContainerDied","Data":"acd1488cbfa23d332ab757fcb3367d0201f1e13328f4f3aaddc7382a6cfd90e9"} Sep 14 11:10:21.585859 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.585844 2556 scope.go:117] "RemoveContainer" containerID="00ec68902b035181ff7079853e9e977fc0d9b2fdd03040c185f90fe1da79595d" Sep 14 11:10:21.593201 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.593187 2556 scope.go:117] "RemoveContainer" containerID="00ec68902b035181ff7079853e9e977fc0d9b2fdd03040c185f90fe1da79595d" Sep 14 11:10:21.593438 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:10:21.593420 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00ec68902b035181ff7079853e9e977fc0d9b2fdd03040c185f90fe1da79595d\": container with ID starting with 00ec68902b035181ff7079853e9e977fc0d9b2fdd03040c185f90fe1da79595d not found: ID does not exist" containerID="00ec68902b035181ff7079853e9e977fc0d9b2fdd03040c185f90fe1da79595d" Sep 14 11:10:21.593494 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.593448 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00ec68902b035181ff7079853e9e977fc0d9b2fdd03040c185f90fe1da79595d"} err="failed to get container status \"00ec68902b035181ff7079853e9e977fc0d9b2fdd03040c185f90fe1da79595d\": rpc error: code = NotFound desc = could not find container \"00ec68902b035181ff7079853e9e977fc0d9b2fdd03040c185f90fe1da79595d\": container with ID starting with 00ec68902b035181ff7079853e9e977fc0d9b2fdd03040c185f90fe1da79595d not found: ID does not exist" Sep 14 11:10:21.644072 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.644048 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4khkp\" (UniqueName: \"kubernetes.io/projected/80ee3da5-767b-4c8c-a93e-38021cbbda56-kube-api-access-4khkp\") pod \"80ee3da5-767b-4c8c-a93e-38021cbbda56\" (UID: \"80ee3da5-767b-4c8c-a93e-38021cbbda56\") " Sep 14 11:10:21.644216 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.644109 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/80ee3da5-767b-4c8c-a93e-38021cbbda56-ready\") pod \"80ee3da5-767b-4c8c-a93e-38021cbbda56\" (UID: \"80ee3da5-767b-4c8c-a93e-38021cbbda56\") " Sep 14 11:10:21.644216 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.644148 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/80ee3da5-767b-4c8c-a93e-38021cbbda56-cni-sysctl-allowlist\") pod \"80ee3da5-767b-4c8c-a93e-38021cbbda56\" (UID: \"80ee3da5-767b-4c8c-a93e-38021cbbda56\") " Sep 14 11:10:21.644216 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.644165 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80ee3da5-767b-4c8c-a93e-38021cbbda56-tuning-conf-dir\") pod \"80ee3da5-767b-4c8c-a93e-38021cbbda56\" (UID: \"80ee3da5-767b-4c8c-a93e-38021cbbda56\") " Sep 14 11:10:21.644373 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.644235 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/80ee3da5-767b-4c8c-a93e-38021cbbda56-tuning-conf-dir" (OuterVolumeSpecName: "tuning-conf-dir") pod "80ee3da5-767b-4c8c-a93e-38021cbbda56" (UID: "80ee3da5-767b-4c8c-a93e-38021cbbda56"). InnerVolumeSpecName "tuning-conf-dir". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 14 11:10:21.644494 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.644387 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ee3da5-767b-4c8c-a93e-38021cbbda56-ready" (OuterVolumeSpecName: "ready") pod "80ee3da5-767b-4c8c-a93e-38021cbbda56" (UID: "80ee3da5-767b-4c8c-a93e-38021cbbda56"). InnerVolumeSpecName "ready". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:10:21.644494 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.644410 2556 reconciler_common.go:299] "Volume detached for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80ee3da5-767b-4c8c-a93e-38021cbbda56-tuning-conf-dir\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:10:21.644494 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.644429 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ee3da5-767b-4c8c-a93e-38021cbbda56-cni-sysctl-allowlist" (OuterVolumeSpecName: "cni-sysctl-allowlist") pod "80ee3da5-767b-4c8c-a93e-38021cbbda56" (UID: "80ee3da5-767b-4c8c-a93e-38021cbbda56"). InnerVolumeSpecName "cni-sysctl-allowlist". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:10:21.645129 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.645069 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5bd5697d46-8dr5t" podStartSLOduration=2.327384541 podStartE2EDuration="2.645055001s" totalImagesPullingTime="317.67046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:10:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:10:21.614288349 +0000 UTC m=+579.677691258" watchObservedRunningTime="2026-09-14 11:10:21.645055001 +0000 UTC m=+579.708457911" Sep 14 11:10:21.646318 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.646296 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ee3da5-767b-4c8c-a93e-38021cbbda56-kube-api-access-4khkp" (OuterVolumeSpecName: "kube-api-access-4khkp") pod "80ee3da5-767b-4c8c-a93e-38021cbbda56" (UID: "80ee3da5-767b-4c8c-a93e-38021cbbda56"). InnerVolumeSpecName "kube-api-access-4khkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:10:21.646403 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.646299 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f97b8968-sbpck"] Sep 14 11:10:21.646512 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.646481 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5f97b8968-sbpck" podUID="59dfb79b-6e56-4307-80cf-443e4ea81b75" containerName="authorino" containerID="cri-o://b02cc8cab9bd67b1918d25e37f25ae904f0540af212731ee0d60260fe864eef2" gracePeriod=30 Sep 14 11:10:21.745644 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.745573 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4khkp\" (UniqueName: \"kubernetes.io/projected/80ee3da5-767b-4c8c-a93e-38021cbbda56-kube-api-access-4khkp\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:10:21.745644 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.745600 2556 reconciler_common.go:299] "Volume detached for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/80ee3da5-767b-4c8c-a93e-38021cbbda56-ready\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:10:21.745644 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.745614 2556 reconciler_common.go:299] "Volume detached for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/80ee3da5-767b-4c8c-a93e-38021cbbda56-cni-sysctl-allowlist\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:10:21.871903 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.871880 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f97b8968-sbpck" Sep 14 11:10:21.911302 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.911278 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-lwh65"] Sep 14 11:10:21.923392 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:21.923372 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-lwh65"] Sep 14 11:10:22.048259 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:22.048196 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pb7wq\" (UniqueName: \"kubernetes.io/projected/59dfb79b-6e56-4307-80cf-443e4ea81b75-kube-api-access-pb7wq\") pod \"59dfb79b-6e56-4307-80cf-443e4ea81b75\" (UID: \"59dfb79b-6e56-4307-80cf-443e4ea81b75\") " Sep 14 11:10:22.049949 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:22.049922 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59dfb79b-6e56-4307-80cf-443e4ea81b75-kube-api-access-pb7wq" (OuterVolumeSpecName: "kube-api-access-pb7wq") pod "59dfb79b-6e56-4307-80cf-443e4ea81b75" (UID: "59dfb79b-6e56-4307-80cf-443e4ea81b75"). InnerVolumeSpecName "kube-api-access-pb7wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:10:22.148776 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:22.148753 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb7wq\" (UniqueName: \"kubernetes.io/projected/59dfb79b-6e56-4307-80cf-443e4ea81b75-kube-api-access-pb7wq\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:10:22.381816 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:22.381785 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80ee3da5-767b-4c8c-a93e-38021cbbda56" path="/var/lib/kubelet/pods/80ee3da5-767b-4c8c-a93e-38021cbbda56/volumes" Sep 14 11:10:22.382109 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:22.382081 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b434ee77-d506-4948-8c88-56526fa1a419" path="/var/lib/kubelet/pods/b434ee77-d506-4948-8c88-56526fa1a419/volumes" Sep 14 11:10:22.591841 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:22.591808 2556 generic.go:356] "Generic (PLEG): container finished" podID="59dfb79b-6e56-4307-80cf-443e4ea81b75" containerID="b02cc8cab9bd67b1918d25e37f25ae904f0540af212731ee0d60260fe864eef2" exitCode=0 Sep 14 11:10:22.592273 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:22.591852 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f97b8968-sbpck" Sep 14 11:10:22.592273 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:22.591885 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f97b8968-sbpck" event={"ID":"59dfb79b-6e56-4307-80cf-443e4ea81b75","Type":"ContainerDied","Data":"b02cc8cab9bd67b1918d25e37f25ae904f0540af212731ee0d60260fe864eef2"} Sep 14 11:10:22.592273 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:22.591917 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f97b8968-sbpck" event={"ID":"59dfb79b-6e56-4307-80cf-443e4ea81b75","Type":"ContainerDied","Data":"49230a4a6aa3753877ba4857c2ab159afa09264a021baf66fe32785a97d7bb65"} Sep 14 11:10:22.592273 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:22.591934 2556 scope.go:117] "RemoveContainer" containerID="b02cc8cab9bd67b1918d25e37f25ae904f0540af212731ee0d60260fe864eef2" Sep 14 11:10:22.599501 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:22.599484 2556 scope.go:117] "RemoveContainer" containerID="b02cc8cab9bd67b1918d25e37f25ae904f0540af212731ee0d60260fe864eef2" Sep 14 11:10:22.599751 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:10:22.599733 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b02cc8cab9bd67b1918d25e37f25ae904f0540af212731ee0d60260fe864eef2\": container with ID starting with b02cc8cab9bd67b1918d25e37f25ae904f0540af212731ee0d60260fe864eef2 not found: ID does not exist" containerID="b02cc8cab9bd67b1918d25e37f25ae904f0540af212731ee0d60260fe864eef2" Sep 14 11:10:22.599801 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:22.599767 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b02cc8cab9bd67b1918d25e37f25ae904f0540af212731ee0d60260fe864eef2"} err="failed to get container status \"b02cc8cab9bd67b1918d25e37f25ae904f0540af212731ee0d60260fe864eef2\": rpc error: code = NotFound desc = could not find container \"b02cc8cab9bd67b1918d25e37f25ae904f0540af212731ee0d60260fe864eef2\": container with ID starting with b02cc8cab9bd67b1918d25e37f25ae904f0540af212731ee0d60260fe864eef2 not found: ID does not exist" Sep 14 11:10:22.624824 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:22.624800 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f97b8968-sbpck"] Sep 14 11:10:22.636211 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:22.636160 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f97b8968-sbpck"] Sep 14 11:10:24.381909 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:10:24.381871 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59dfb79b-6e56-4307-80cf-443e4ea81b75" path="/var/lib/kubelet/pods/59dfb79b-6e56-4307-80cf-443e4ea81b75/volumes" Sep 14 11:11:13.151669 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:13.151635 2556 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-fdw2s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.23:15021/healthz/ready\": read tcp 10.132.0.2:51426->10.132.0.23:15021: read: connection reset by peer" start-of-body= Sep 14 11:11:13.152017 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:13.151704 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" podUID="3afd8f26-9ef1-47db-ac05-349b80a4c6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.23:15021/healthz/ready\": read tcp 10.132.0.2:51426->10.132.0.23:15021: read: connection reset by peer" Sep 14 11:11:13.816931 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:13.816901 2556 generic.go:356] "Generic (PLEG): container finished" podID="3afd8f26-9ef1-47db-ac05-349b80a4c6b1" containerID="8417d2fa10c6a652b7194b39c3e7d53a313740a7c3ce1b5dcf2800cdb9262d36" exitCode=0 Sep 14 11:11:13.817106 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:13.816971 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" event={"ID":"3afd8f26-9ef1-47db-ac05-349b80a4c6b1","Type":"ContainerDied","Data":"8417d2fa10c6a652b7194b39c3e7d53a313740a7c3ce1b5dcf2800cdb9262d36"} Sep 14 11:11:13.817351 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:13.817335 2556 scope.go:117] "RemoveContainer" containerID="8417d2fa10c6a652b7194b39c3e7d53a313740a7c3ce1b5dcf2800cdb9262d36" Sep 14 11:11:13.818228 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:13.818198 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:11:13.818319 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:13.818278 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:11:13.818378 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:13.818335 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:11:14.823578 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:14.823547 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" event={"ID":"3afd8f26-9ef1-47db-ac05-349b80a4c6b1","Type":"ContainerStarted","Data":"0baf2a6b5003054125fd2863e91bdec4195189edff900535b7bd2ac779d59201"} Sep 14 11:11:14.942281 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:14.942244 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:11:14.942490 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:14.942468 2556 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-fdw2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" start-of-body= Sep 14 11:11:14.942561 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:14.942525 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" podUID="3afd8f26-9ef1-47db-ac05-349b80a4c6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" Sep 14 11:11:15.942364 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:15.942329 2556 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-fdw2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" start-of-body= Sep 14 11:11:15.942737 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:15.942390 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" podUID="3afd8f26-9ef1-47db-ac05-349b80a4c6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" Sep 14 11:11:16.942562 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:16.942526 2556 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-fdw2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" start-of-body= Sep 14 11:11:16.942919 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:16.942576 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" podUID="3afd8f26-9ef1-47db-ac05-349b80a4c6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" Sep 14 11:11:17.942407 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:17.942369 2556 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-fdw2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" start-of-body= Sep 14 11:11:17.942570 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:17.942427 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" podUID="3afd8f26-9ef1-47db-ac05-349b80a4c6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" Sep 14 11:11:18.942853 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:18.942818 2556 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-fdw2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" start-of-body= Sep 14 11:11:18.943267 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:18.942878 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" podUID="3afd8f26-9ef1-47db-ac05-349b80a4c6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" Sep 14 11:11:19.945969 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:19.945933 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:11:19.946370 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:19.946273 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:11:19.946959 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:19.946940 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fdw2s" Sep 14 11:11:26.669225 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.669191 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r"] Sep 14 11:11:26.669634 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.669483 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59dfb79b-6e56-4307-80cf-443e4ea81b75" containerName="authorino" Sep 14 11:11:26.669634 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.669499 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="59dfb79b-6e56-4307-80cf-443e4ea81b75" containerName="authorino" Sep 14 11:11:26.669634 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.669511 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b434ee77-d506-4948-8c88-56526fa1a419" containerName="authorino" Sep 14 11:11:26.669634 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.669516 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b434ee77-d506-4948-8c88-56526fa1a419" containerName="authorino" Sep 14 11:11:26.669634 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.669525 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80ee3da5-767b-4c8c-a93e-38021cbbda56" containerName="kube-multus-additional-cni-plugins" Sep 14 11:11:26.669634 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.669534 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="80ee3da5-767b-4c8c-a93e-38021cbbda56" containerName="kube-multus-additional-cni-plugins" Sep 14 11:11:26.669634 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.669594 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="80ee3da5-767b-4c8c-a93e-38021cbbda56" containerName="kube-multus-additional-cni-plugins" Sep 14 11:11:26.669634 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.669602 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="59dfb79b-6e56-4307-80cf-443e4ea81b75" containerName="authorino" Sep 14 11:11:26.669634 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.669611 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b434ee77-d506-4948-8c88-56526fa1a419" containerName="authorino" Sep 14 11:11:26.671647 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.671622 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" Sep 14 11:11:26.675112 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.675081 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 14 11:11:26.676037 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.676019 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 14 11:11:26.676341 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.676324 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6r78c\"" Sep 14 11:11:26.676425 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.676349 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 14 11:11:26.687658 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.687634 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r"] Sep 14 11:11:26.795656 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.795621 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e62bde0-fa42-4ac3-83b4-74e7bd0fec04-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r\" (UID: \"8e62bde0-fa42-4ac3-83b4-74e7bd0fec04\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" Sep 14 11:11:26.795824 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.795668 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e62bde0-fa42-4ac3-83b4-74e7bd0fec04-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r\" (UID: \"8e62bde0-fa42-4ac3-83b4-74e7bd0fec04\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" Sep 14 11:11:26.795824 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.795709 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e62bde0-fa42-4ac3-83b4-74e7bd0fec04-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r\" (UID: \"8e62bde0-fa42-4ac3-83b4-74e7bd0fec04\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" Sep 14 11:11:26.795824 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.795728 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e62bde0-fa42-4ac3-83b4-74e7bd0fec04-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r\" (UID: \"8e62bde0-fa42-4ac3-83b4-74e7bd0fec04\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" Sep 14 11:11:26.795824 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.795755 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e62bde0-fa42-4ac3-83b4-74e7bd0fec04-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r\" (UID: \"8e62bde0-fa42-4ac3-83b4-74e7bd0fec04\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" Sep 14 11:11:26.795824 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.795790 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgpnr\" (UniqueName: \"kubernetes.io/projected/8e62bde0-fa42-4ac3-83b4-74e7bd0fec04-kube-api-access-sgpnr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r\" (UID: \"8e62bde0-fa42-4ac3-83b4-74e7bd0fec04\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" Sep 14 11:11:26.896474 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.896440 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgpnr\" (UniqueName: \"kubernetes.io/projected/8e62bde0-fa42-4ac3-83b4-74e7bd0fec04-kube-api-access-sgpnr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r\" (UID: \"8e62bde0-fa42-4ac3-83b4-74e7bd0fec04\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" Sep 14 11:11:26.896659 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.896504 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e62bde0-fa42-4ac3-83b4-74e7bd0fec04-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r\" (UID: \"8e62bde0-fa42-4ac3-83b4-74e7bd0fec04\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" Sep 14 11:11:26.896659 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.896549 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e62bde0-fa42-4ac3-83b4-74e7bd0fec04-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r\" (UID: \"8e62bde0-fa42-4ac3-83b4-74e7bd0fec04\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" Sep 14 11:11:26.896659 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.896579 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e62bde0-fa42-4ac3-83b4-74e7bd0fec04-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r\" (UID: \"8e62bde0-fa42-4ac3-83b4-74e7bd0fec04\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" Sep 14 11:11:26.896659 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.896601 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e62bde0-fa42-4ac3-83b4-74e7bd0fec04-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r\" (UID: \"8e62bde0-fa42-4ac3-83b4-74e7bd0fec04\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" Sep 14 11:11:26.896659 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.896627 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e62bde0-fa42-4ac3-83b4-74e7bd0fec04-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r\" (UID: \"8e62bde0-fa42-4ac3-83b4-74e7bd0fec04\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" Sep 14 11:11:26.896951 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.896929 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e62bde0-fa42-4ac3-83b4-74e7bd0fec04-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r\" (UID: \"8e62bde0-fa42-4ac3-83b4-74e7bd0fec04\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" Sep 14 11:11:26.897008 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.896985 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e62bde0-fa42-4ac3-83b4-74e7bd0fec04-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r\" (UID: \"8e62bde0-fa42-4ac3-83b4-74e7bd0fec04\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" Sep 14 11:11:26.897042 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.897030 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e62bde0-fa42-4ac3-83b4-74e7bd0fec04-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r\" (UID: \"8e62bde0-fa42-4ac3-83b4-74e7bd0fec04\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" Sep 14 11:11:26.898858 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.898839 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e62bde0-fa42-4ac3-83b4-74e7bd0fec04-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r\" (UID: \"8e62bde0-fa42-4ac3-83b4-74e7bd0fec04\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" Sep 14 11:11:26.899194 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.899174 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e62bde0-fa42-4ac3-83b4-74e7bd0fec04-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r\" (UID: \"8e62bde0-fa42-4ac3-83b4-74e7bd0fec04\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" Sep 14 11:11:26.905977 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.905953 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgpnr\" (UniqueName: \"kubernetes.io/projected/8e62bde0-fa42-4ac3-83b4-74e7bd0fec04-kube-api-access-sgpnr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r\" (UID: \"8e62bde0-fa42-4ac3-83b4-74e7bd0fec04\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" Sep 14 11:11:26.981248 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:26.981157 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" Sep 14 11:11:27.260981 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:27.260889 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r"] Sep 14 11:11:27.264342 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:11:27.264317 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e62bde0_fa42_4ac3_83b4_74e7bd0fec04.slice/crio-1f18804c0343aeeedadc7fc6b12e1c59521bb85e9b6640ab9ac3b19eb351f81a WatchSource:0}: Error finding container 1f18804c0343aeeedadc7fc6b12e1c59521bb85e9b6640ab9ac3b19eb351f81a: Status 404 returned error can't find the container with id 1f18804c0343aeeedadc7fc6b12e1c59521bb85e9b6640ab9ac3b19eb351f81a Sep 14 11:11:27.265773 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:27.265756 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 11:11:27.878589 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:27.878545 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" event={"ID":"8e62bde0-fa42-4ac3-83b4-74e7bd0fec04","Type":"ContainerStarted","Data":"1f18804c0343aeeedadc7fc6b12e1c59521bb85e9b6640ab9ac3b19eb351f81a"} Sep 14 11:11:28.883877 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:28.883838 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" event={"ID":"8e62bde0-fa42-4ac3-83b4-74e7bd0fec04","Type":"ContainerStarted","Data":"a1621c86b92d7a1d21220a770904a53a55c76a44c1c577400efd356aa27fb717"} Sep 14 11:11:28.907911 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:28.907860 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" podStartSLOduration=1.7481597309999999 podStartE2EDuration="2.907842436s" totalImagesPullingTime="1.159682705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:11:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:11:28.906304344 +0000 UTC m=+646.969707254" watchObservedRunningTime="2026-09-14 11:11:28.907842436 +0000 UTC m=+646.971245345" Sep 14 11:11:36.982248 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:36.982215 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" Sep 14 11:11:36.982820 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:36.982304 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" Sep 14 11:11:36.994283 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:36.994260 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" Sep 14 11:11:37.937293 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:11:37.937265 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r" Sep 14 11:12:01.006290 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:01.006254 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4"] Sep 14 11:12:01.008562 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:01.008546 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" Sep 14 11:12:01.011364 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:01.011342 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 14 11:12:01.042105 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:01.042070 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4"] Sep 14 11:12:01.169777 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:01.169734 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e6ad956-e4b7-4c58-b145-486a8e935797-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gggh4\" (UID: \"7e6ad956-e4b7-4c58-b145-486a8e935797\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" Sep 14 11:12:01.169777 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:01.169775 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e6ad956-e4b7-4c58-b145-486a8e935797-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gggh4\" (UID: \"7e6ad956-e4b7-4c58-b145-486a8e935797\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" Sep 14 11:12:01.169993 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:01.169799 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e6ad956-e4b7-4c58-b145-486a8e935797-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gggh4\" (UID: \"7e6ad956-e4b7-4c58-b145-486a8e935797\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" Sep 14 11:12:01.169993 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:01.169825 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e6ad956-e4b7-4c58-b145-486a8e935797-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gggh4\" (UID: \"7e6ad956-e4b7-4c58-b145-486a8e935797\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" Sep 14 11:12:01.169993 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:01.169853 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvkzg\" (UniqueName: \"kubernetes.io/projected/7e6ad956-e4b7-4c58-b145-486a8e935797-kube-api-access-kvkzg\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gggh4\" (UID: \"7e6ad956-e4b7-4c58-b145-486a8e935797\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" Sep 14 11:12:01.169993 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:01.169930 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e6ad956-e4b7-4c58-b145-486a8e935797-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gggh4\" (UID: \"7e6ad956-e4b7-4c58-b145-486a8e935797\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" Sep 14 11:12:01.270398 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:01.270315 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e6ad956-e4b7-4c58-b145-486a8e935797-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gggh4\" (UID: \"7e6ad956-e4b7-4c58-b145-486a8e935797\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" Sep 14 11:12:01.270398 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:01.270373 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvkzg\" (UniqueName: \"kubernetes.io/projected/7e6ad956-e4b7-4c58-b145-486a8e935797-kube-api-access-kvkzg\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gggh4\" (UID: \"7e6ad956-e4b7-4c58-b145-486a8e935797\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" Sep 14 11:12:01.270619 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:01.270409 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e6ad956-e4b7-4c58-b145-486a8e935797-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gggh4\" (UID: \"7e6ad956-e4b7-4c58-b145-486a8e935797\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" Sep 14 11:12:01.270619 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:01.270444 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e6ad956-e4b7-4c58-b145-486a8e935797-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gggh4\" (UID: \"7e6ad956-e4b7-4c58-b145-486a8e935797\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" Sep 14 11:12:01.270619 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:01.270488 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e6ad956-e4b7-4c58-b145-486a8e935797-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gggh4\" (UID: \"7e6ad956-e4b7-4c58-b145-486a8e935797\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" Sep 14 11:12:01.270619 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:01.270511 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e6ad956-e4b7-4c58-b145-486a8e935797-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gggh4\" (UID: \"7e6ad956-e4b7-4c58-b145-486a8e935797\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" Sep 14 11:12:01.270819 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:01.270735 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e6ad956-e4b7-4c58-b145-486a8e935797-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gggh4\" (UID: \"7e6ad956-e4b7-4c58-b145-486a8e935797\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" Sep 14 11:12:01.270819 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:01.270774 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e6ad956-e4b7-4c58-b145-486a8e935797-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gggh4\" (UID: \"7e6ad956-e4b7-4c58-b145-486a8e935797\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" Sep 14 11:12:01.270920 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:01.270849 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e6ad956-e4b7-4c58-b145-486a8e935797-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gggh4\" (UID: \"7e6ad956-e4b7-4c58-b145-486a8e935797\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" Sep 14 11:12:01.272590 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:01.272566 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e6ad956-e4b7-4c58-b145-486a8e935797-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gggh4\" (UID: \"7e6ad956-e4b7-4c58-b145-486a8e935797\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" Sep 14 11:12:01.272889 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:01.272871 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e6ad956-e4b7-4c58-b145-486a8e935797-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gggh4\" (UID: \"7e6ad956-e4b7-4c58-b145-486a8e935797\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" Sep 14 11:12:01.278791 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:01.278768 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvkzg\" (UniqueName: \"kubernetes.io/projected/7e6ad956-e4b7-4c58-b145-486a8e935797-kube-api-access-kvkzg\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gggh4\" (UID: \"7e6ad956-e4b7-4c58-b145-486a8e935797\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" Sep 14 11:12:01.319330 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:01.319303 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" Sep 14 11:12:01.613705 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:01.613674 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4"] Sep 14 11:12:01.617037 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:12:01.616998 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e6ad956_e4b7_4c58_b145_486a8e935797.slice/crio-d4f60e4991d3a7bfc80b72f56adb0596d9f96d7341540ddbdefaa0090617af41 WatchSource:0}: Error finding container d4f60e4991d3a7bfc80b72f56adb0596d9f96d7341540ddbdefaa0090617af41: Status 404 returned error can't find the container with id d4f60e4991d3a7bfc80b72f56adb0596d9f96d7341540ddbdefaa0090617af41 Sep 14 11:12:02.023972 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:02.023896 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" event={"ID":"7e6ad956-e4b7-4c58-b145-486a8e935797","Type":"ContainerStarted","Data":"922783a3198861687e9c935c62fad256d2a027421e523d09d58804757acb8299"} Sep 14 11:12:02.023972 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:02.023946 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" event={"ID":"7e6ad956-e4b7-4c58-b145-486a8e935797","Type":"ContainerStarted","Data":"d4f60e4991d3a7bfc80b72f56adb0596d9f96d7341540ddbdefaa0090617af41"} Sep 14 11:12:02.055504 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:02.055455 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" podStartSLOduration=1.8752390110000001 podStartE2EDuration="2.055440597s" totalImagesPullingTime="180.201586ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:12:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:12:02.054167623 +0000 UTC m=+680.117570535" watchObservedRunningTime="2026-09-14 11:12:02.055440597 +0000 UTC m=+680.118843506" Sep 14 11:12:11.319505 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:11.319468 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" Sep 14 11:12:11.319948 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:11.319527 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" Sep 14 11:12:11.331751 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:11.331729 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" Sep 14 11:12:12.075370 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:12:12.075344 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gggh4" Sep 14 11:13:18.383134 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:18.383102 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8"] Sep 14 11:13:18.385434 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:18.385416 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" Sep 14 11:13:18.400692 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:18.400665 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8"] Sep 14 11:13:18.400805 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:18.400671 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 14 11:13:18.536313 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:18.536273 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc132696-2458-49ad-8b2e-75edf35e07f0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8\" (UID: \"dc132696-2458-49ad-8b2e-75edf35e07f0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" Sep 14 11:13:18.536521 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:18.536319 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc132696-2458-49ad-8b2e-75edf35e07f0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8\" (UID: \"dc132696-2458-49ad-8b2e-75edf35e07f0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" Sep 14 11:13:18.536521 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:18.536442 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc132696-2458-49ad-8b2e-75edf35e07f0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8\" (UID: \"dc132696-2458-49ad-8b2e-75edf35e07f0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" Sep 14 11:13:18.536521 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:18.536481 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc132696-2458-49ad-8b2e-75edf35e07f0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8\" (UID: \"dc132696-2458-49ad-8b2e-75edf35e07f0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" Sep 14 11:13:18.536521 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:18.536506 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbnq4\" (UniqueName: \"kubernetes.io/projected/dc132696-2458-49ad-8b2e-75edf35e07f0-kube-api-access-zbnq4\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8\" (UID: \"dc132696-2458-49ad-8b2e-75edf35e07f0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" Sep 14 11:13:18.536696 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:18.536543 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc132696-2458-49ad-8b2e-75edf35e07f0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8\" (UID: \"dc132696-2458-49ad-8b2e-75edf35e07f0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" Sep 14 11:13:18.637277 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:18.637184 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc132696-2458-49ad-8b2e-75edf35e07f0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8\" (UID: \"dc132696-2458-49ad-8b2e-75edf35e07f0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" Sep 14 11:13:18.637277 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:18.637228 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc132696-2458-49ad-8b2e-75edf35e07f0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8\" (UID: \"dc132696-2458-49ad-8b2e-75edf35e07f0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" Sep 14 11:13:18.637492 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:18.637359 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc132696-2458-49ad-8b2e-75edf35e07f0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8\" (UID: \"dc132696-2458-49ad-8b2e-75edf35e07f0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" Sep 14 11:13:18.637492 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:18.637395 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc132696-2458-49ad-8b2e-75edf35e07f0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8\" (UID: \"dc132696-2458-49ad-8b2e-75edf35e07f0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" Sep 14 11:13:18.637492 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:18.637420 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbnq4\" (UniqueName: \"kubernetes.io/projected/dc132696-2458-49ad-8b2e-75edf35e07f0-kube-api-access-zbnq4\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8\" (UID: \"dc132696-2458-49ad-8b2e-75edf35e07f0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" Sep 14 11:13:18.637492 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:18.637456 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc132696-2458-49ad-8b2e-75edf35e07f0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8\" (UID: \"dc132696-2458-49ad-8b2e-75edf35e07f0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" Sep 14 11:13:18.637700 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:18.637641 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc132696-2458-49ad-8b2e-75edf35e07f0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8\" (UID: \"dc132696-2458-49ad-8b2e-75edf35e07f0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" Sep 14 11:13:18.637859 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:18.637829 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc132696-2458-49ad-8b2e-75edf35e07f0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8\" (UID: \"dc132696-2458-49ad-8b2e-75edf35e07f0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" Sep 14 11:13:18.637859 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:18.637858 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc132696-2458-49ad-8b2e-75edf35e07f0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8\" (UID: \"dc132696-2458-49ad-8b2e-75edf35e07f0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" Sep 14 11:13:18.639649 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:18.639622 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc132696-2458-49ad-8b2e-75edf35e07f0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8\" (UID: \"dc132696-2458-49ad-8b2e-75edf35e07f0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" Sep 14 11:13:18.639994 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:18.639976 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc132696-2458-49ad-8b2e-75edf35e07f0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8\" (UID: \"dc132696-2458-49ad-8b2e-75edf35e07f0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" Sep 14 11:13:18.647651 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:18.647633 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbnq4\" (UniqueName: \"kubernetes.io/projected/dc132696-2458-49ad-8b2e-75edf35e07f0-kube-api-access-zbnq4\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8\" (UID: \"dc132696-2458-49ad-8b2e-75edf35e07f0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" Sep 14 11:13:18.695628 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:18.695598 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" Sep 14 11:13:18.966652 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:18.966606 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8"] Sep 14 11:13:18.968695 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:13:18.968673 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc132696_2458_49ad_8b2e_75edf35e07f0.slice/crio-333555e1baca154d31de06fbd07083f0d4113df85d9ffc726da69cfc40c05b5d WatchSource:0}: Error finding container 333555e1baca154d31de06fbd07083f0d4113df85d9ffc726da69cfc40c05b5d: Status 404 returned error can't find the container with id 333555e1baca154d31de06fbd07083f0d4113df85d9ffc726da69cfc40c05b5d Sep 14 11:13:19.338527 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:19.338451 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" event={"ID":"dc132696-2458-49ad-8b2e-75edf35e07f0","Type":"ContainerStarted","Data":"f1fbf1e5e50ad2dd7d7f02537e4973d648de617740f312f277d63ace08f1cbe2"} Sep 14 11:13:19.338527 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:19.338502 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" event={"ID":"dc132696-2458-49ad-8b2e-75edf35e07f0","Type":"ContainerStarted","Data":"333555e1baca154d31de06fbd07083f0d4113df85d9ffc726da69cfc40c05b5d"} Sep 14 11:13:19.359369 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:19.359323 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" podStartSLOduration=1.180105764 podStartE2EDuration="1.359309111s" totalImagesPullingTime="179.203347ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:13:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:13:19.357644099 +0000 UTC m=+757.421047009" watchObservedRunningTime="2026-09-14 11:13:19.359309111 +0000 UTC m=+757.422712076" Sep 14 11:13:28.696650 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:28.696611 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" Sep 14 11:13:28.697134 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:28.696692 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" Sep 14 11:13:28.708761 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:28.708739 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" Sep 14 11:13:29.389283 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:13:29.389256 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8" Sep 14 11:15:02.600271 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:02.600237 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29823075-mvksl"] Sep 14 11:15:02.602456 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:02.602434 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823075-mvksl" Sep 14 11:15:02.605129 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:02.605108 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f98vt\"" Sep 14 11:15:02.611632 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:02.611615 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823075-mvksl" Sep 14 11:15:02.625944 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:02.625920 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29823075-mvksl"] Sep 14 11:15:02.887859 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:02.887838 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29823075-mvksl"] Sep 14 11:15:02.889416 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:15:02.889382 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda081176_ae4e_434f_a22b_c5652e07661a.slice/crio-7dd37618ab734cd1dd92e6eae1e6cf8a07c28769718d445e35e37278056aeed7 WatchSource:0}: Error finding container 7dd37618ab734cd1dd92e6eae1e6cf8a07c28769718d445e35e37278056aeed7: Status 404 returned error can't find the container with id 7dd37618ab734cd1dd92e6eae1e6cf8a07c28769718d445e35e37278056aeed7 Sep 14 11:15:03.752836 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:03.752800 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823075-mvksl" event={"ID":"da081176-ae4e-434f-a22b-c5652e07661a","Type":"ContainerStarted","Data":"7dd37618ab734cd1dd92e6eae1e6cf8a07c28769718d445e35e37278056aeed7"} Sep 14 11:15:05.762801 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:05.762750 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823075-mvksl" event={"ID":"da081176-ae4e-434f-a22b-c5652e07661a","Type":"ContainerStarted","Data":"2004bad9f77d161543b42d552e14eb8070f2d8c3c8b89c97a7fba671137fe087"} Sep 14 11:15:06.768442 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:06.768409 2556 generic.go:356] "Generic (PLEG): container finished" podID="da081176-ae4e-434f-a22b-c5652e07661a" containerID="2004bad9f77d161543b42d552e14eb8070f2d8c3c8b89c97a7fba671137fe087" exitCode=0 Sep 14 11:15:06.768773 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:06.768490 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823075-mvksl" event={"ID":"da081176-ae4e-434f-a22b-c5652e07661a","Type":"ContainerDied","Data":"2004bad9f77d161543b42d552e14eb8070f2d8c3c8b89c97a7fba671137fe087"} Sep 14 11:15:07.891548 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:07.891527 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823075-mvksl" Sep 14 11:15:08.778617 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:08.778552 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823075-mvksl" event={"ID":"da081176-ae4e-434f-a22b-c5652e07661a","Type":"ContainerDied","Data":"7dd37618ab734cd1dd92e6eae1e6cf8a07c28769718d445e35e37278056aeed7"} Sep 14 11:15:08.778617 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:08.778603 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dd37618ab734cd1dd92e6eae1e6cf8a07c28769718d445e35e37278056aeed7" Sep 14 11:15:08.778617 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:08.778572 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823075-mvksl" Sep 14 11:15:17.841858 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:17.841812 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d66bbffcf-cd2bk"] Sep 14 11:15:17.842236 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:17.842102 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da081176-ae4e-434f-a22b-c5652e07661a" containerName="cleanup" Sep 14 11:15:17.842236 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:17.842115 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="da081176-ae4e-434f-a22b-c5652e07661a" containerName="cleanup" Sep 14 11:15:17.842236 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:17.842178 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="da081176-ae4e-434f-a22b-c5652e07661a" containerName="cleanup" Sep 14 11:15:17.844003 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:17.843979 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d66bbffcf-cd2bk" Sep 14 11:15:17.863649 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:17.863626 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d66bbffcf-cd2bk"] Sep 14 11:15:17.958937 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:17.958912 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cckn6\" (UniqueName: \"kubernetes.io/projected/551680a3-3886-4d7a-8f07-1a5683b4b4d3-kube-api-access-cckn6\") pod \"authorino-d66bbffcf-cd2bk\" (UID: \"551680a3-3886-4d7a-8f07-1a5683b4b4d3\") " pod="kuadrant-system/authorino-d66bbffcf-cd2bk" Sep 14 11:15:17.959054 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:17.958951 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/551680a3-3886-4d7a-8f07-1a5683b4b4d3-tls-cert\") pod \"authorino-d66bbffcf-cd2bk\" (UID: \"551680a3-3886-4d7a-8f07-1a5683b4b4d3\") " pod="kuadrant-system/authorino-d66bbffcf-cd2bk" Sep 14 11:15:18.059289 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:18.059263 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/551680a3-3886-4d7a-8f07-1a5683b4b4d3-tls-cert\") pod \"authorino-d66bbffcf-cd2bk\" (UID: \"551680a3-3886-4d7a-8f07-1a5683b4b4d3\") " pod="kuadrant-system/authorino-d66bbffcf-cd2bk" Sep 14 11:15:18.059389 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:18.059318 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cckn6\" (UniqueName: \"kubernetes.io/projected/551680a3-3886-4d7a-8f07-1a5683b4b4d3-kube-api-access-cckn6\") pod \"authorino-d66bbffcf-cd2bk\" (UID: \"551680a3-3886-4d7a-8f07-1a5683b4b4d3\") " pod="kuadrant-system/authorino-d66bbffcf-cd2bk" Sep 14 11:15:18.061665 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:18.061644 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/551680a3-3886-4d7a-8f07-1a5683b4b4d3-tls-cert\") pod \"authorino-d66bbffcf-cd2bk\" (UID: \"551680a3-3886-4d7a-8f07-1a5683b4b4d3\") " pod="kuadrant-system/authorino-d66bbffcf-cd2bk" Sep 14 11:15:18.068339 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:18.068315 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cckn6\" (UniqueName: \"kubernetes.io/projected/551680a3-3886-4d7a-8f07-1a5683b4b4d3-kube-api-access-cckn6\") pod \"authorino-d66bbffcf-cd2bk\" (UID: \"551680a3-3886-4d7a-8f07-1a5683b4b4d3\") " pod="kuadrant-system/authorino-d66bbffcf-cd2bk" Sep 14 11:15:18.152783 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:18.152761 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d66bbffcf-cd2bk" Sep 14 11:15:18.445599 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:18.445522 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d66bbffcf-cd2bk"] Sep 14 11:15:18.448604 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:15:18.448578 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod551680a3_3886_4d7a_8f07_1a5683b4b4d3.slice/crio-350b549a8ff5264c8d2a7bf9229c40706a313e0e6fc982944ab4a0989e4954d2 WatchSource:0}: Error finding container 350b549a8ff5264c8d2a7bf9229c40706a313e0e6fc982944ab4a0989e4954d2: Status 404 returned error can't find the container with id 350b549a8ff5264c8d2a7bf9229c40706a313e0e6fc982944ab4a0989e4954d2 Sep 14 11:15:18.819789 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:18.819708 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d66bbffcf-cd2bk" event={"ID":"551680a3-3886-4d7a-8f07-1a5683b4b4d3","Type":"ContainerStarted","Data":"350b549a8ff5264c8d2a7bf9229c40706a313e0e6fc982944ab4a0989e4954d2"} Sep 14 11:15:19.827147 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:19.827101 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d66bbffcf-cd2bk" event={"ID":"551680a3-3886-4d7a-8f07-1a5683b4b4d3","Type":"ContainerStarted","Data":"a2684071382001fe0b51c0d0547c9504a87dd96e35fbbc82d8f47b79ddac5073"} Sep 14 11:15:19.929752 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:19.929681 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-d66bbffcf-cd2bk" podStartSLOduration=2.299352248 podStartE2EDuration="2.929655664s" totalImagesPullingTime="630.303416ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:15:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:15:19.869638047 +0000 UTC m=+877.933040958" watchObservedRunningTime="2026-09-14 11:15:19.929655664 +0000 UTC m=+877.993058575" Sep 14 11:15:19.931327 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:19.931297 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5bd5697d46-8dr5t"] Sep 14 11:15:19.931802 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:19.931562 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5bd5697d46-8dr5t" podUID="92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e" containerName="authorino" containerID="cri-o://289fbc516258695866ad0a5f723635ffb37d9ad441654618bb1edcf9ad8e78a4" gracePeriod=30 Sep 14 11:15:20.176348 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:20.176322 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bd5697d46-8dr5t" Sep 14 11:15:20.278056 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:20.278027 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bm52\" (UniqueName: \"kubernetes.io/projected/92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e-kube-api-access-9bm52\") pod \"92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e\" (UID: \"92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e\") " Sep 14 11:15:20.278221 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:20.278123 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e-tls-cert\") pod \"92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e\" (UID: \"92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e\") " Sep 14 11:15:20.280242 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:20.280204 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e-kube-api-access-9bm52" (OuterVolumeSpecName: "kube-api-access-9bm52") pod "92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e" (UID: "92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e"). InnerVolumeSpecName "kube-api-access-9bm52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:15:20.294284 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:20.292265 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e" (UID: "92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:15:20.384065 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:20.380363 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bm52\" (UniqueName: \"kubernetes.io/projected/92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e-kube-api-access-9bm52\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:15:20.384065 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:20.380399 2556 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e-tls-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:15:20.835450 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:20.835368 2556 generic.go:356] "Generic (PLEG): container finished" podID="92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e" containerID="289fbc516258695866ad0a5f723635ffb37d9ad441654618bb1edcf9ad8e78a4" exitCode=0 Sep 14 11:15:20.835450 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:20.835441 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bd5697d46-8dr5t" Sep 14 11:15:20.835938 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:20.835457 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bd5697d46-8dr5t" event={"ID":"92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e","Type":"ContainerDied","Data":"289fbc516258695866ad0a5f723635ffb37d9ad441654618bb1edcf9ad8e78a4"} Sep 14 11:15:20.835938 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:20.835507 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bd5697d46-8dr5t" event={"ID":"92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e","Type":"ContainerDied","Data":"dde403831314c17a5a94f30c5bcf807ab28542b891caea598263c102dc1d6ee4"} Sep 14 11:15:20.835938 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:20.835529 2556 scope.go:117] "RemoveContainer" containerID="289fbc516258695866ad0a5f723635ffb37d9ad441654618bb1edcf9ad8e78a4" Sep 14 11:15:20.843846 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:20.843831 2556 scope.go:117] "RemoveContainer" containerID="289fbc516258695866ad0a5f723635ffb37d9ad441654618bb1edcf9ad8e78a4" Sep 14 11:15:20.844146 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:15:20.844124 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"289fbc516258695866ad0a5f723635ffb37d9ad441654618bb1edcf9ad8e78a4\": container with ID starting with 289fbc516258695866ad0a5f723635ffb37d9ad441654618bb1edcf9ad8e78a4 not found: ID does not exist" containerID="289fbc516258695866ad0a5f723635ffb37d9ad441654618bb1edcf9ad8e78a4" Sep 14 11:15:20.844275 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:20.844149 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"289fbc516258695866ad0a5f723635ffb37d9ad441654618bb1edcf9ad8e78a4"} err="failed to get container status \"289fbc516258695866ad0a5f723635ffb37d9ad441654618bb1edcf9ad8e78a4\": rpc error: code = NotFound desc = could not find container \"289fbc516258695866ad0a5f723635ffb37d9ad441654618bb1edcf9ad8e78a4\": container with ID starting with 289fbc516258695866ad0a5f723635ffb37d9ad441654618bb1edcf9ad8e78a4 not found: ID does not exist" Sep 14 11:15:20.857039 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:20.857015 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5bd5697d46-8dr5t"] Sep 14 11:15:20.861629 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:20.861605 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5bd5697d46-8dr5t"] Sep 14 11:15:22.381176 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:22.381150 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e" path="/var/lib/kubelet/pods/92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e/volumes" Sep 14 11:15:37.219645 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.219568 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt"] Sep 14 11:15:37.219984 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.219851 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e" containerName="authorino" Sep 14 11:15:37.219984 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.219862 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e" containerName="authorino" Sep 14 11:15:37.219984 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.219915 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="92ae7246-1fa8-4ee4-aae3-6b51b2f00d0e" containerName="authorino" Sep 14 11:15:37.223162 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.223142 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.225890 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.225873 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-de0f7a-openshift-default-dockercfg-b4tm9\"" Sep 14 11:15:37.241328 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.241307 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt"] Sep 14 11:15:37.304625 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.304597 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-istio-data\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.304745 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.304647 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-credential-socket\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.304745 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.304711 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39b3e46-7504-46f1-8c15-9ad48b95123b-istiod-ca-cert\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.304836 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.304788 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-istio-envoy\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.304836 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.304818 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-workload-socket\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.304918 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.304849 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqp99\" (UniqueName: \"kubernetes.io/projected/e39b3e46-7504-46f1-8c15-9ad48b95123b-kube-api-access-qqp99\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.304957 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.304925 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39b3e46-7504-46f1-8c15-9ad48b95123b-istio-podinfo\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.304995 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.304964 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-workload-certs\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.304995 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.304982 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39b3e46-7504-46f1-8c15-9ad48b95123b-istio-token\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.313158 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.313137 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g"] Sep 14 11:15:37.315531 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.315515 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.327819 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.327803 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-6f3d0f-openshift-default-dockercfg-sxgdg\"" Sep 14 11:15:37.348820 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.348799 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g"] Sep 14 11:15:37.406005 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.405981 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-workload-certs\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.406137 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.406015 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b95e55c-aadc-440b-af1e-ce128f283433-istiod-ca-cert\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.406137 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.406046 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39b3e46-7504-46f1-8c15-9ad48b95123b-istio-token\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.406252 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.406149 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-istio-data\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.406252 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.406178 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b95e55c-aadc-440b-af1e-ce128f283433-istio-podinfo\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.406252 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.406213 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-credential-socket\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.406252 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.406238 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39b3e46-7504-46f1-8c15-9ad48b95123b-istiod-ca-cert\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.406417 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.406281 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-istio-envoy\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.406459 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.406405 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-istio-data\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.406459 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.406452 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-workload-socket\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.406531 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.406454 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-workload-certs\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.406531 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.406490 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-istio-envoy\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.406531 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.406508 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-credential-socket\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.406531 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.406521 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-credential-socket\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.406731 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.406557 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-istio-data\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.406731 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.406592 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqp99\" (UniqueName: \"kubernetes.io/projected/e39b3e46-7504-46f1-8c15-9ad48b95123b-kube-api-access-qqp99\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.406731 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.406626 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlqf4\" (UniqueName: \"kubernetes.io/projected/0b95e55c-aadc-440b-af1e-ce128f283433-kube-api-access-tlqf4\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.406731 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.406685 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-workload-socket\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.406731 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.406690 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-workload-socket\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.406731 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.406715 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39b3e46-7504-46f1-8c15-9ad48b95123b-istio-podinfo\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.406960 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.406744 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-workload-certs\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.406960 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.406768 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b95e55c-aadc-440b-af1e-ce128f283433-istio-token\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.406960 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.406910 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39b3e46-7504-46f1-8c15-9ad48b95123b-istiod-ca-cert\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.408727 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.408681 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-istio-envoy\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.409062 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.409043 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39b3e46-7504-46f1-8c15-9ad48b95123b-istio-podinfo\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.421537 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.421514 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39b3e46-7504-46f1-8c15-9ad48b95123b-istio-token\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.426850 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.426823 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqp99\" (UniqueName: \"kubernetes.io/projected/e39b3e46-7504-46f1-8c15-9ad48b95123b-kube-api-access-qqp99\") pod \"e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.453577 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.453547 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr"] Sep 14 11:15:37.456434 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.456416 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.458810 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.458793 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6d272c-openshift-default-dockercfg-xkff2\"" Sep 14 11:15:37.478649 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.478595 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr"] Sep 14 11:15:37.507963 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.507938 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b95e55c-aadc-440b-af1e-ce128f283433-istio-podinfo\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.508065 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.507979 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-istio-data\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.508065 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.507999 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-istio-envoy\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.508065 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.508025 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-credential-socket\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.508065 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.508057 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlqf4\" (UniqueName: \"kubernetes.io/projected/0b95e55c-aadc-440b-af1e-ce128f283433-kube-api-access-tlqf4\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.508306 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.508115 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-workload-socket\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.508306 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.508149 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-workload-certs\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.508306 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.508172 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b95e55c-aadc-440b-af1e-ce128f283433-istio-token\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.508306 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.508199 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b95e55c-aadc-440b-af1e-ce128f283433-istiod-ca-cert\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.508495 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.508387 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-credential-socket\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.508495 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.508418 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-istio-data\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.508637 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.508617 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-workload-certs\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.508689 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.508645 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-workload-socket\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.508854 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.508840 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b95e55c-aadc-440b-af1e-ce128f283433-istiod-ca-cert\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.510170 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.510151 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-istio-envoy\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.510237 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.510182 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b95e55c-aadc-440b-af1e-ce128f283433-istio-podinfo\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.527530 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.527506 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlqf4\" (UniqueName: \"kubernetes.io/projected/0b95e55c-aadc-440b-af1e-ce128f283433-kube-api-access-tlqf4\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.535924 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.535905 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b95e55c-aadc-440b-af1e-ce128f283433-istio-token\") pod \"e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.554914 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.554892 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:37.608819 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.608693 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-istio-envoy\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.608819 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.608738 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f43dd7d-23a3-4749-a795-562f8a28497b-istio-token\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.608999 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.608819 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-istio-data\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.608999 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.608888 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f43dd7d-23a3-4749-a795-562f8a28497b-istio-podinfo\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.608999 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.608924 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-credential-socket\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.608999 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.608951 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-workload-certs\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.608999 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.608978 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-workload-socket\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.609271 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.609062 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl5b9\" (UniqueName: \"kubernetes.io/projected/9f43dd7d-23a3-4749-a795-562f8a28497b-kube-api-access-nl5b9\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.609271 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.609138 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f43dd7d-23a3-4749-a795-562f8a28497b-istiod-ca-cert\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.624621 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.624596 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:37.709596 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.709563 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-credential-socket\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.709750 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.709612 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-workload-certs\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.709750 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.709643 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-workload-socket\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.709750 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.709679 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl5b9\" (UniqueName: \"kubernetes.io/projected/9f43dd7d-23a3-4749-a795-562f8a28497b-kube-api-access-nl5b9\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.709750 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.709724 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f43dd7d-23a3-4749-a795-562f8a28497b-istiod-ca-cert\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.709962 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.709777 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-istio-envoy\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.709962 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.709810 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f43dd7d-23a3-4749-a795-562f8a28497b-istio-token\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.709962 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.709844 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-istio-data\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.709962 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.709875 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f43dd7d-23a3-4749-a795-562f8a28497b-istio-podinfo\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.710162 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.709995 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-credential-socket\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.710162 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.710074 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-workload-certs\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.710361 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.710342 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-istio-data\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.710446 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.710424 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-workload-socket\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.710523 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.710502 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f43dd7d-23a3-4749-a795-562f8a28497b-istiod-ca-cert\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.712047 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.712029 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-istio-envoy\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.712200 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.712184 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f43dd7d-23a3-4749-a795-562f8a28497b-istio-podinfo\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.745505 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.745455 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f43dd7d-23a3-4749-a795-562f8a28497b-istio-token\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.745918 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.745901 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl5b9\" (UniqueName: \"kubernetes.io/projected/9f43dd7d-23a3-4749-a795-562f8a28497b-kube-api-access-nl5b9\") pod \"e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.768106 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.768064 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:37.848626 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.848601 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt"] Sep 14 11:15:37.850723 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:15:37.850702 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39b3e46_7504_46f1_8c15_9ad48b95123b.slice/crio-7a7af31112ce032b70436a2a0fff1a7d1e41cf19b9c69d8e9a348cee14a8fc17 WatchSource:0}: Error finding container 7a7af31112ce032b70436a2a0fff1a7d1e41cf19b9c69d8e9a348cee14a8fc17: Status 404 returned error can't find the container with id 7a7af31112ce032b70436a2a0fff1a7d1e41cf19b9c69d8e9a348cee14a8fc17 Sep 14 11:15:37.852813 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.852778 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:15:37.852903 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.852843 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:15:37.852903 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.852870 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:15:37.911632 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.911605 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" event={"ID":"e39b3e46-7504-46f1-8c15-9ad48b95123b","Type":"ContainerStarted","Data":"dd46bac4364093ca5a2c7e23d6a097cba6df4c9c227ef85f3d8eb2c317336a2a"} Sep 14 11:15:37.911712 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.911652 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" event={"ID":"e39b3e46-7504-46f1-8c15-9ad48b95123b","Type":"ContainerStarted","Data":"7a7af31112ce032b70436a2a0fff1a7d1e41cf19b9c69d8e9a348cee14a8fc17"} Sep 14 11:15:37.930267 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.930242 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g"] Sep 14 11:15:37.930990 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:15:37.930952 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b95e55c_aadc_440b_af1e_ce128f283433.slice/crio-dcae9f88a136896dab8d90ae42051ceebd634de957520fbf24edbf3213adf959 WatchSource:0}: Error finding container dcae9f88a136896dab8d90ae42051ceebd634de957520fbf24edbf3213adf959: Status 404 returned error can't find the container with id dcae9f88a136896dab8d90ae42051ceebd634de957520fbf24edbf3213adf959 Sep 14 11:15:37.933049 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.933022 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:15:37.933127 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.933078 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:15:37.933162 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.933123 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:15:37.956490 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:37.956440 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" podStartSLOduration=0.956420972 podStartE2EDuration="956.420972ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:15:37.952742226 +0000 UTC m=+896.016145136" watchObservedRunningTime="2026-09-14 11:15:37.956420972 +0000 UTC m=+896.019823885" Sep 14 11:15:38.055611 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:38.055531 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr"] Sep 14 11:15:38.058724 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:15:38.058691 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f43dd7d_23a3_4749_a795_562f8a28497b.slice/crio-ddc5b2db624689dd5901f52aae6d39a7ab69e29c7a46dc5b9084091a0c536421 WatchSource:0}: Error finding container ddc5b2db624689dd5901f52aae6d39a7ab69e29c7a46dc5b9084091a0c536421: Status 404 returned error can't find the container with id ddc5b2db624689dd5901f52aae6d39a7ab69e29c7a46dc5b9084091a0c536421 Sep 14 11:15:38.061597 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:38.061564 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:15:38.061690 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:38.061639 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:15:38.061690 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:38.061682 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:15:38.555253 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:38.555227 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:38.556392 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:38.556366 2556 patch_prober.go:28] interesting pod/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Sep 14 11:15:38.556502 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:38.556414 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" podUID="e39b3e46-7504-46f1-8c15-9ad48b95123b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Sep 14 11:15:38.917174 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:38.917137 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" event={"ID":"9f43dd7d-23a3-4749-a795-562f8a28497b","Type":"ContainerStarted","Data":"7006ec3d7f921d10d9dba6ee22d646ec37c25132e8e74e9bb36d010e552a4005"} Sep 14 11:15:38.917367 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:38.917207 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" event={"ID":"9f43dd7d-23a3-4749-a795-562f8a28497b","Type":"ContainerStarted","Data":"ddc5b2db624689dd5901f52aae6d39a7ab69e29c7a46dc5b9084091a0c536421"} Sep 14 11:15:38.918872 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:38.918845 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" event={"ID":"0b95e55c-aadc-440b-af1e-ce128f283433","Type":"ContainerStarted","Data":"b2cc998f408459b66ac6db3633cffe9f711be7c6e06c20fbf8468065c734d9b3"} Sep 14 11:15:38.918991 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:38.918896 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" event={"ID":"0b95e55c-aadc-440b-af1e-ce128f283433","Type":"ContainerStarted","Data":"dcae9f88a136896dab8d90ae42051ceebd634de957520fbf24edbf3213adf959"} Sep 14 11:15:39.032159 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:39.032117 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" podStartSLOduration=2.032104219 podStartE2EDuration="2.032104219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:15:39.030661548 +0000 UTC m=+897.094064457" watchObservedRunningTime="2026-09-14 11:15:39.032104219 +0000 UTC m=+897.095507119" Sep 14 11:15:39.145989 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:39.145943 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" podStartSLOduration=2.14592932 podStartE2EDuration="2.14592932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:15:39.14298628 +0000 UTC m=+897.206389191" watchObservedRunningTime="2026-09-14 11:15:39.14592932 +0000 UTC m=+897.209332229" Sep 14 11:15:39.559235 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:39.559208 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:39.625065 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:39.625038 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:39.629453 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:39.629434 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:39.768840 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:39.768816 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:39.773124 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:39.773081 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:39.923415 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:39.923385 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:39.923577 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:39.923436 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:39.923577 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:39.923460 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:39.923984 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:39.923959 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:15:39.924156 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:39.924136 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:39.924205 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:39.924189 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:15:46.171778 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.171733 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd"] Sep 14 11:15:46.178445 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.178423 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.185979 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.185387 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-27a1a2-openshift-default-dockercfg-qc52l\"" Sep 14 11:15:46.204486 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.204456 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd"] Sep 14 11:15:46.286573 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.286532 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af40be6-bb3e-472d-bec5-eaf171cc900d-istio-podinfo\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.286767 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.286611 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-workload-certs\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.286767 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.286654 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af40be6-bb3e-472d-bec5-eaf171cc900d-istiod-ca-cert\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.286767 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.286683 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-istio-data\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.286947 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.286812 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47jjl\" (UniqueName: \"kubernetes.io/projected/3af40be6-bb3e-472d-bec5-eaf171cc900d-kube-api-access-47jjl\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.286947 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.286873 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af40be6-bb3e-472d-bec5-eaf171cc900d-istio-token\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.286947 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.286908 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-workload-socket\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.287132 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.286973 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-credential-socket\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.287132 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.287022 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-istio-envoy\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.389494 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.389454 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af40be6-bb3e-472d-bec5-eaf171cc900d-istio-podinfo\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.389696 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.389514 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-workload-certs\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.389696 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.389546 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af40be6-bb3e-472d-bec5-eaf171cc900d-istiod-ca-cert\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.389696 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.389573 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-istio-data\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.389696 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.389629 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47jjl\" (UniqueName: \"kubernetes.io/projected/3af40be6-bb3e-472d-bec5-eaf171cc900d-kube-api-access-47jjl\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.389696 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.389664 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af40be6-bb3e-472d-bec5-eaf171cc900d-istio-token\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.389696 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.389691 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-workload-socket\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.390028 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.389718 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-credential-socket\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.390028 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.389747 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-istio-envoy\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.390752 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.390502 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-workload-certs\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.392008 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.391107 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af40be6-bb3e-472d-bec5-eaf171cc900d-istiod-ca-cert\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.392008 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.391354 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-istio-data\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.392008 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.391568 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-workload-socket\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.392246 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.392012 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-credential-socket\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.393975 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.393913 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af40be6-bb3e-472d-bec5-eaf171cc900d-istio-podinfo\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.396177 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.396154 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-istio-envoy\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.406011 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.405983 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47jjl\" (UniqueName: \"kubernetes.io/projected/3af40be6-bb3e-472d-bec5-eaf171cc900d-kube-api-access-47jjl\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.407142 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.407119 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af40be6-bb3e-472d-bec5-eaf171cc900d-istio-token\") pod \"e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.490004 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.489909 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:46.821347 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.821257 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd"] Sep 14 11:15:46.826238 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:15:46.826195 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af40be6_bb3e_472d_bec5_eaf171cc900d.slice/crio-bd216b59173fb983118c79197ae94fea53182d1047333587c2bf8be5a81ccee7 WatchSource:0}: Error finding container bd216b59173fb983118c79197ae94fea53182d1047333587c2bf8be5a81ccee7: Status 404 returned error can't find the container with id bd216b59173fb983118c79197ae94fea53182d1047333587c2bf8be5a81ccee7 Sep 14 11:15:46.829306 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.829266 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:15:46.829408 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.829356 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:15:46.829408 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.829401 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:15:46.961344 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.961304 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" event={"ID":"3af40be6-bb3e-472d-bec5-eaf171cc900d","Type":"ContainerStarted","Data":"771f7c18cedea400923dde3d268a2a384bd887588ff68bbd82f6c4acec9b591d"} Sep 14 11:15:46.961513 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.961369 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" event={"ID":"3af40be6-bb3e-472d-bec5-eaf171cc900d","Type":"ContainerStarted","Data":"bd216b59173fb983118c79197ae94fea53182d1047333587c2bf8be5a81ccee7"} Sep 14 11:15:46.996835 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:46.996774 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" podStartSLOduration=0.99675334 podStartE2EDuration="996.75334ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:15:46.996512664 +0000 UTC m=+905.059915575" watchObservedRunningTime="2026-09-14 11:15:46.99675334 +0000 UTC m=+905.060156248" Sep 14 11:15:47.496263 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:47.496225 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:47.502110 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:47.498208 2556 patch_prober.go:28] interesting pod/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Sep 14 11:15:47.502110 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:47.498267 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" podUID="3af40be6-bb3e-472d-bec5-eaf171cc900d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Sep 14 11:15:48.496645 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:48.496614 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:48.969647 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:48.969607 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:48.970708 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:48.970684 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:15:55.007706 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:55.007317 2556 patch_prober.go:28] interesting pod/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": read tcp 10.132.0.2:42930->10.132.0.34:15021: read: connection reset by peer" start-of-body= Sep 14 11:15:55.007706 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:55.007393 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" podUID="e39b3e46-7504-46f1-8c15-9ad48b95123b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": read tcp 10.132.0.2:42930->10.132.0.34:15021: read: connection reset by peer" Sep 14 11:15:56.012040 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:56.011991 2556 generic.go:356] "Generic (PLEG): container finished" podID="e39b3e46-7504-46f1-8c15-9ad48b95123b" containerID="dd46bac4364093ca5a2c7e23d6a097cba6df4c9c227ef85f3d8eb2c317336a2a" exitCode=0 Sep 14 11:15:56.012678 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:56.012637 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" event={"ID":"e39b3e46-7504-46f1-8c15-9ad48b95123b","Type":"ContainerDied","Data":"dd46bac4364093ca5a2c7e23d6a097cba6df4c9c227ef85f3d8eb2c317336a2a"} Sep 14 11:15:56.013351 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:56.013333 2556 scope.go:117] "RemoveContainer" containerID="dd46bac4364093ca5a2c7e23d6a097cba6df4c9c227ef85f3d8eb2c317336a2a" Sep 14 11:15:56.015643 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:56.015613 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:15:56.015837 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:56.015816 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:15:56.015962 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:56.015946 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:15:57.022871 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:57.021929 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" event={"ID":"e39b3e46-7504-46f1-8c15-9ad48b95123b","Type":"ContainerStarted","Data":"8eaec34685ecabe86146f32a2c3266ea0c60b96ab38fe5b951834972840c4d2e"} Sep 14 11:15:57.555582 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:57.555547 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:15:57.555934 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:57.555909 2556 patch_prober.go:28] interesting pod/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Sep 14 11:15:57.556115 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:57.556069 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" podUID="e39b3e46-7504-46f1-8c15-9ad48b95123b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Sep 14 11:15:58.556007 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:58.555970 2556 patch_prober.go:28] interesting pod/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Sep 14 11:15:58.556620 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:58.556579 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" podUID="e39b3e46-7504-46f1-8c15-9ad48b95123b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Sep 14 11:15:59.555326 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:59.555292 2556 patch_prober.go:28] interesting pod/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Sep 14 11:15:59.555612 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:15:59.555584 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" podUID="e39b3e46-7504-46f1-8c15-9ad48b95123b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Sep 14 11:16:00.556255 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:00.556226 2556 patch_prober.go:28] interesting pod/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Sep 14 11:16:00.556633 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:00.556607 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" podUID="e39b3e46-7504-46f1-8c15-9ad48b95123b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Sep 14 11:16:01.555567 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:01.555482 2556 patch_prober.go:28] interesting pod/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Sep 14 11:16:01.555567 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:01.555554 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" podUID="e39b3e46-7504-46f1-8c15-9ad48b95123b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Sep 14 11:16:02.555339 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:02.555307 2556 patch_prober.go:28] interesting pod/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Sep 14 11:16:02.555805 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:02.555376 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" podUID="e39b3e46-7504-46f1-8c15-9ad48b95123b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Sep 14 11:16:03.559317 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:03.559289 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:16:03.559713 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:03.559651 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:16:03.560370 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:03.560353 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:16:04.062397 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:04.062374 2556 generic.go:356] "Generic (PLEG): container finished" podID="0b95e55c-aadc-440b-af1e-ce128f283433" containerID="b2cc998f408459b66ac6db3633cffe9f711be7c6e06c20fbf8468065c734d9b3" exitCode=0 Sep 14 11:16:04.062538 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:04.062440 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" event={"ID":"0b95e55c-aadc-440b-af1e-ce128f283433","Type":"ContainerDied","Data":"b2cc998f408459b66ac6db3633cffe9f711be7c6e06c20fbf8468065c734d9b3"} Sep 14 11:16:04.062953 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:04.062935 2556 scope.go:117] "RemoveContainer" containerID="b2cc998f408459b66ac6db3633cffe9f711be7c6e06c20fbf8468065c734d9b3" Sep 14 11:16:04.064056 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:04.064027 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:16:04.064158 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:04.064134 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:16:04.064221 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:04.064191 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:16:05.068455 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:05.068422 2556 generic.go:356] "Generic (PLEG): container finished" podID="9f43dd7d-23a3-4749-a795-562f8a28497b" containerID="7006ec3d7f921d10d9dba6ee22d646ec37c25132e8e74e9bb36d010e552a4005" exitCode=0 Sep 14 11:16:05.068931 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:05.068489 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" event={"ID":"9f43dd7d-23a3-4749-a795-562f8a28497b","Type":"ContainerDied","Data":"7006ec3d7f921d10d9dba6ee22d646ec37c25132e8e74e9bb36d010e552a4005"} Sep 14 11:16:05.068994 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:05.068940 2556 scope.go:117] "RemoveContainer" containerID="7006ec3d7f921d10d9dba6ee22d646ec37c25132e8e74e9bb36d010e552a4005" Sep 14 11:16:05.070049 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:05.070014 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:16:05.070166 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:05.070113 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:16:05.070166 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:05.070155 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:16:05.070604 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:05.070585 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" event={"ID":"0b95e55c-aadc-440b-af1e-ce128f283433","Type":"ContainerStarted","Data":"2e9e8bb671637d334bae5b40eaad04fc53c5978a587459be11a37287d680d82c"} Sep 14 11:16:05.625706 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:05.625668 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:16:05.626664 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:05.626643 2556 patch_prober.go:28] interesting pod/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Sep 14 11:16:05.626718 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:05.626699 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" podUID="0b95e55c-aadc-440b-af1e-ce128f283433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Sep 14 11:16:06.077845 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.077755 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" event={"ID":"9f43dd7d-23a3-4749-a795-562f8a28497b","Type":"ContainerStarted","Data":"c5d2f86ecca952cd91bd4209f41ce2ec7020fee9080eaae7d53c2ca8a04a66f1"} Sep 14 11:16:06.625447 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.625408 2556 patch_prober.go:28] interesting pod/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Sep 14 11:16:06.625726 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.625697 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" podUID="0b95e55c-aadc-440b-af1e-ce128f283433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Sep 14 11:16:06.768853 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.768827 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:16:06.769687 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.769659 2556 patch_prober.go:28] interesting pod/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 14 11:16:06.769792 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.769700 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" podUID="9f43dd7d-23a3-4749-a795-562f8a28497b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 14 11:16:06.828000 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.827968 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv"] Sep 14 11:16:06.838378 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.838355 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:06.842794 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.842773 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-39ab34-openshift-default-dockercfg-pvzb4\"" Sep 14 11:16:06.856507 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.855949 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv"] Sep 14 11:16:06.895174 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.891677 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-workload-socket\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:06.895174 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.891756 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0376711b-9ae6-441c-8083-bf6086d85862-istiod-ca-cert\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:06.895174 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.891834 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-istio-data\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:06.895174 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.891866 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-credential-socket\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:06.895174 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.891909 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-workload-certs\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:06.895174 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.891955 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-istio-envoy\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:06.895174 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.891997 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j56rr\" (UniqueName: \"kubernetes.io/projected/0376711b-9ae6-441c-8083-bf6086d85862-kube-api-access-j56rr\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:06.895174 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.892060 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0376711b-9ae6-441c-8083-bf6086d85862-istio-token\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:06.895174 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.892121 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0376711b-9ae6-441c-8083-bf6086d85862-istio-podinfo\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:06.996115 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.995421 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-workload-socket\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:06.996115 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.995477 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0376711b-9ae6-441c-8083-bf6086d85862-istiod-ca-cert\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:06.996115 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.995528 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-istio-data\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:06.996115 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.995556 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-credential-socket\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:06.996115 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.995580 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-workload-certs\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:06.996115 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.995623 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-istio-envoy\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:06.996115 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.995649 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j56rr\" (UniqueName: \"kubernetes.io/projected/0376711b-9ae6-441c-8083-bf6086d85862-kube-api-access-j56rr\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:06.996115 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.995690 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0376711b-9ae6-441c-8083-bf6086d85862-istio-token\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:06.996115 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.995730 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0376711b-9ae6-441c-8083-bf6086d85862-istio-podinfo\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:06.996732 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.996435 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-credential-socket\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:06.996732 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.996651 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-workload-socket\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:06.997687 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.997218 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0376711b-9ae6-441c-8083-bf6086d85862-istiod-ca-cert\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:06.997687 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.997435 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-istio-data\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:06.997929 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:06.997907 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-workload-certs\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:07.018111 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:07.015403 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-istio-envoy\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:07.018111 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:07.015926 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0376711b-9ae6-441c-8083-bf6086d85862-istio-podinfo\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:07.027237 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:07.027202 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j56rr\" (UniqueName: \"kubernetes.io/projected/0376711b-9ae6-441c-8083-bf6086d85862-kube-api-access-j56rr\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:07.032112 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:07.029433 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0376711b-9ae6-441c-8083-bf6086d85862-istio-token\") pod \"e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:07.157905 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:07.157803 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:07.522058 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:07.521963 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv"] Sep 14 11:16:07.530919 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:07.529510 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:16:07.530919 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:07.529596 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:16:07.530919 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:07.529640 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:16:07.625744 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:07.625703 2556 patch_prober.go:28] interesting pod/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Sep 14 11:16:07.625889 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:07.625767 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" podUID="0b95e55c-aadc-440b-af1e-ce128f283433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Sep 14 11:16:07.769254 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:07.769221 2556 patch_prober.go:28] interesting pod/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 14 11:16:07.769427 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:07.769318 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" podUID="9f43dd7d-23a3-4749-a795-562f8a28497b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 14 11:16:08.092620 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:08.092498 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" event={"ID":"0376711b-9ae6-441c-8083-bf6086d85862","Type":"ContainerStarted","Data":"02b41bc73bd9922b833a9dd9ecc282df26b728a7c95764b6c351f864804e946f"} Sep 14 11:16:08.092620 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:08.092567 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" event={"ID":"0376711b-9ae6-441c-8083-bf6086d85862","Type":"ContainerStarted","Data":"a257d8121793002187a89d0013ed10fa149d34b43e70d55e750dca041718d6e5"} Sep 14 11:16:08.159523 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:08.159486 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:08.160547 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:08.160513 2556 patch_prober.go:28] interesting pod/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Sep 14 11:16:08.160658 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:08.160584 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" podUID="0376711b-9ae6-441c-8083-bf6086d85862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Sep 14 11:16:08.625702 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:08.625657 2556 patch_prober.go:28] interesting pod/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Sep 14 11:16:08.625997 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:08.625955 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" podUID="0b95e55c-aadc-440b-af1e-ce128f283433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Sep 14 11:16:08.768699 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:08.768664 2556 patch_prober.go:28] interesting pod/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 14 11:16:08.769004 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:08.768953 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" podUID="9f43dd7d-23a3-4749-a795-562f8a28497b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 14 11:16:09.167509 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:09.167478 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:09.212477 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:09.212419 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" podStartSLOduration=3.212397008 podStartE2EDuration="3.212397008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:16:08.145441434 +0000 UTC m=+926.208844344" watchObservedRunningTime="2026-09-14 11:16:09.212397008 +0000 UTC m=+927.275799919" Sep 14 11:16:09.625772 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:09.625736 2556 patch_prober.go:28] interesting pod/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Sep 14 11:16:09.626074 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:09.626032 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" podUID="0b95e55c-aadc-440b-af1e-ce128f283433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Sep 14 11:16:09.769157 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:09.769122 2556 patch_prober.go:28] interesting pod/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 14 11:16:09.769333 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:09.769181 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" podUID="9f43dd7d-23a3-4749-a795-562f8a28497b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 14 11:16:09.923952 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:09.923844 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:16:09.925036 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:09.925012 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:16:10.106592 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:10.106565 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:10.111757 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:10.111735 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:10.767625 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:10.767590 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:16:10.768362 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:10.768339 2556 patch_prober.go:28] interesting pod/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 14 11:16:10.768462 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:10.768386 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" podUID="9f43dd7d-23a3-4749-a795-562f8a28497b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 14 11:16:10.769292 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:10.769278 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:16:11.772805 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:11.772776 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:16:11.773828 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:11.773809 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:16:16.137406 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:16.137373 2556 generic.go:356] "Generic (PLEG): container finished" podID="3af40be6-bb3e-472d-bec5-eaf171cc900d" containerID="771f7c18cedea400923dde3d268a2a384bd887588ff68bbd82f6c4acec9b591d" exitCode=0 Sep 14 11:16:16.137829 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:16.137446 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" event={"ID":"3af40be6-bb3e-472d-bec5-eaf171cc900d","Type":"ContainerDied","Data":"771f7c18cedea400923dde3d268a2a384bd887588ff68bbd82f6c4acec9b591d"} Sep 14 11:16:16.137882 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:16.137871 2556 scope.go:117] "RemoveContainer" containerID="771f7c18cedea400923dde3d268a2a384bd887588ff68bbd82f6c4acec9b591d" Sep 14 11:16:16.138929 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:16.138887 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:16:16.139037 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:16.139010 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:16:16.139114 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:16.139067 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:16:17.143317 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:17.143283 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" event={"ID":"3af40be6-bb3e-472d-bec5-eaf171cc900d","Type":"ContainerStarted","Data":"492c8b18b062c7b56820bedf91066cdc959d9799c62045c95a4dc11a639b97cf"} Sep 14 11:16:17.491702 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:17.491607 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:16:17.492057 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:17.492033 2556 patch_prober.go:28] interesting pod/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Sep 14 11:16:17.492267 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:17.492242 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" podUID="3af40be6-bb3e-472d-bec5-eaf171cc900d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Sep 14 11:16:18.490583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:18.490544 2556 patch_prober.go:28] interesting pod/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Sep 14 11:16:18.491182 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:18.491150 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" podUID="3af40be6-bb3e-472d-bec5-eaf171cc900d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Sep 14 11:16:18.969940 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:18.969899 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:16:19.490623 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:19.490588 2556 patch_prober.go:28] interesting pod/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Sep 14 11:16:19.491298 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:19.491267 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" podUID="3af40be6-bb3e-472d-bec5-eaf171cc900d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Sep 14 11:16:20.490824 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:20.490784 2556 patch_prober.go:28] interesting pod/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Sep 14 11:16:20.491437 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:20.491402 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" podUID="3af40be6-bb3e-472d-bec5-eaf171cc900d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Sep 14 11:16:21.495495 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:21.495459 2556 patch_prober.go:28] interesting pod/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Sep 14 11:16:21.496059 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:21.496027 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" podUID="3af40be6-bb3e-472d-bec5-eaf171cc900d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Sep 14 11:16:22.425510 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.425478 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j"] Sep 14 11:16:22.427907 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.427886 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.430448 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.430425 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1ac71380-openshift-default-dockercfg-xs865\"" Sep 14 11:16:22.447828 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.447805 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j"] Sep 14 11:16:22.492127 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.491507 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-credential-socket\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.492127 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.491552 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-istio-envoy\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.492127 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.491579 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-istio-data\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.492127 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.491618 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-workload-socket\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.492127 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.491652 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a034a7a3-4e1d-479f-bc00-14c64bc82031-istio-token\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.492127 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.491676 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a034a7a3-4e1d-479f-bc00-14c64bc82031-istiod-ca-cert\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.492127 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.491712 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcz42\" (UniqueName: \"kubernetes.io/projected/a034a7a3-4e1d-479f-bc00-14c64bc82031-kube-api-access-qcz42\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.492127 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.491795 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a034a7a3-4e1d-479f-bc00-14c64bc82031-istio-podinfo\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.492127 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.491838 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-workload-certs\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.507354 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.507331 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:16:22.516344 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.516315 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:16:22.596030 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.595992 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a034a7a3-4e1d-479f-bc00-14c64bc82031-istio-podinfo\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.596316 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.596294 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-workload-certs\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.596456 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.596440 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-credential-socket\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.596578 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.596562 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-istio-envoy\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.596688 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.596671 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-istio-data\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.596828 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.596813 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-workload-socket\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.596943 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.596929 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a034a7a3-4e1d-479f-bc00-14c64bc82031-istio-token\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.597049 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.597035 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a034a7a3-4e1d-479f-bc00-14c64bc82031-istiod-ca-cert\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.597179 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.597163 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcz42\" (UniqueName: \"kubernetes.io/projected/a034a7a3-4e1d-479f-bc00-14c64bc82031-kube-api-access-qcz42\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.598499 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.598475 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-istio-data\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.599059 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.599037 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-workload-certs\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.599451 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.599425 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-credential-socket\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.603105 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.599907 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-workload-socket\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.603105 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.600584 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a034a7a3-4e1d-479f-bc00-14c64bc82031-istiod-ca-cert\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.608544 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.608328 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-istio-envoy\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.622163 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.621609 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a034a7a3-4e1d-479f-bc00-14c64bc82031-istio-podinfo\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.625961 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.625933 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a034a7a3-4e1d-479f-bc00-14c64bc82031-istio-token\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.626452 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.626432 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcz42\" (UniqueName: \"kubernetes.io/projected/a034a7a3-4e1d-479f-bc00-14c64bc82031-kube-api-access-qcz42\") pod \"e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:22.738343 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:22.738073 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:23.040287 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:23.040213 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j"] Sep 14 11:16:23.049267 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:16:23.049243 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda034a7a3_4e1d_479f_bc00_14c64bc82031.slice/crio-006dacd1f7e8d40f2de29900fa67e7101dd2624072c8a6fa0c24f1f9d4df53d0 WatchSource:0}: Error finding container 006dacd1f7e8d40f2de29900fa67e7101dd2624072c8a6fa0c24f1f9d4df53d0: Status 404 returned error can't find the container with id 006dacd1f7e8d40f2de29900fa67e7101dd2624072c8a6fa0c24f1f9d4df53d0 Sep 14 11:16:23.051412 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:23.051377 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:16:23.051504 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:23.051457 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:16:23.051562 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:23.051502 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:16:23.174150 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:23.174118 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" event={"ID":"a034a7a3-4e1d-479f-bc00-14c64bc82031","Type":"ContainerStarted","Data":"3e1f4c28e50c6130e0a5542168adefe62d00080476d8b6062b444b18358df40c"} Sep 14 11:16:23.174308 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:23.174168 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" event={"ID":"a034a7a3-4e1d-479f-bc00-14c64bc82031","Type":"ContainerStarted","Data":"006dacd1f7e8d40f2de29900fa67e7101dd2624072c8a6fa0c24f1f9d4df53d0"} Sep 14 11:16:23.739057 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:23.739029 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:24.744537 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:24.744505 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:24.774601 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:24.774551 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" podStartSLOduration=2.774533357 podStartE2EDuration="2.774533357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:16:23.203849773 +0000 UTC m=+941.267252684" watchObservedRunningTime="2026-09-14 11:16:24.774533357 +0000 UTC m=+942.837936266" Sep 14 11:16:25.184369 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:25.184338 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:25.190797 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:25.190757 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:30.115953 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.115917 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz"] Sep 14 11:16:30.118674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.118649 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz" Sep 14 11:16:30.145911 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.145885 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1ac71380-serving-cert\"" Sep 14 11:16:30.146201 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.146166 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 11:16:30.146823 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.146809 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 14 11:16:30.147001 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.146980 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w7q69\"" Sep 14 11:16:30.151426 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.151409 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 14 11:16:30.172458 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.172432 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b091ec2-5232-4776-8fdd-4029894256a1-openshift-service-ca\") pod \"maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz\" (UID: \"6b091ec2-5232-4776-8fdd-4029894256a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz" Sep 14 11:16:30.172544 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.172487 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b091ec2-5232-4776-8fdd-4029894256a1-maas-api-tls\") pod \"maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz\" (UID: \"6b091ec2-5232-4776-8fdd-4029894256a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz" Sep 14 11:16:30.172590 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.172577 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b091ec2-5232-4776-8fdd-4029894256a1-metrics-certs\") pod \"maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz\" (UID: \"6b091ec2-5232-4776-8fdd-4029894256a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz" Sep 14 11:16:30.172625 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.172594 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4f2f\" (UniqueName: \"kubernetes.io/projected/6b091ec2-5232-4776-8fdd-4029894256a1-kube-api-access-j4f2f\") pod \"maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz\" (UID: \"6b091ec2-5232-4776-8fdd-4029894256a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz" Sep 14 11:16:30.176274 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.176249 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz"] Sep 14 11:16:30.217420 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.217383 2556 generic.go:356] "Generic (PLEG): container finished" podID="0376711b-9ae6-441c-8083-bf6086d85862" containerID="02b41bc73bd9922b833a9dd9ecc282df26b728a7c95764b6c351f864804e946f" exitCode=0 Sep 14 11:16:30.217574 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.217455 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" event={"ID":"0376711b-9ae6-441c-8083-bf6086d85862","Type":"ContainerDied","Data":"02b41bc73bd9922b833a9dd9ecc282df26b728a7c95764b6c351f864804e946f"} Sep 14 11:16:30.217877 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.217863 2556 scope.go:117] "RemoveContainer" containerID="02b41bc73bd9922b833a9dd9ecc282df26b728a7c95764b6c351f864804e946f" Sep 14 11:16:30.219126 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.219076 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:16:30.219220 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.219175 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:16:30.219265 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.219223 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:16:30.273499 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.273475 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b091ec2-5232-4776-8fdd-4029894256a1-openshift-service-ca\") pod \"maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz\" (UID: \"6b091ec2-5232-4776-8fdd-4029894256a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz" Sep 14 11:16:30.273626 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.273597 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b091ec2-5232-4776-8fdd-4029894256a1-maas-api-tls\") pod \"maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz\" (UID: \"6b091ec2-5232-4776-8fdd-4029894256a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz" Sep 14 11:16:30.273690 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.273636 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b091ec2-5232-4776-8fdd-4029894256a1-metrics-certs\") pod \"maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz\" (UID: \"6b091ec2-5232-4776-8fdd-4029894256a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz" Sep 14 11:16:30.273690 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.273664 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4f2f\" (UniqueName: \"kubernetes.io/projected/6b091ec2-5232-4776-8fdd-4029894256a1-kube-api-access-j4f2f\") pod \"maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz\" (UID: \"6b091ec2-5232-4776-8fdd-4029894256a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz" Sep 14 11:16:30.274321 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.274298 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b091ec2-5232-4776-8fdd-4029894256a1-openshift-service-ca\") pod \"maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz\" (UID: \"6b091ec2-5232-4776-8fdd-4029894256a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz" Sep 14 11:16:30.276680 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.276640 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b091ec2-5232-4776-8fdd-4029894256a1-metrics-certs\") pod \"maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz\" (UID: \"6b091ec2-5232-4776-8fdd-4029894256a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz" Sep 14 11:16:30.276784 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.276746 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b091ec2-5232-4776-8fdd-4029894256a1-maas-api-tls\") pod \"maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz\" (UID: \"6b091ec2-5232-4776-8fdd-4029894256a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz" Sep 14 11:16:30.292239 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.292198 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4f2f\" (UniqueName: \"kubernetes.io/projected/6b091ec2-5232-4776-8fdd-4029894256a1-kube-api-access-j4f2f\") pod \"maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz\" (UID: \"6b091ec2-5232-4776-8fdd-4029894256a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz" Sep 14 11:16:30.428334 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.428242 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz" Sep 14 11:16:30.709155 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.709128 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz"] Sep 14 11:16:30.716354 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:30.714048 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 11:16:31.226826 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:31.226787 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" event={"ID":"0376711b-9ae6-441c-8083-bf6086d85862","Type":"ContainerStarted","Data":"0333120bed9ae7af0196867688767c0b79012e8ba95361397c7db974e826746b"} Sep 14 11:16:31.228555 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:31.228519 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz" event={"ID":"6b091ec2-5232-4776-8fdd-4029894256a1","Type":"ContainerStarted","Data":"ddf0edfb8832aa6c4638355e8191aa7202ac48d12b82e22eb3123b7b23fce72a"} Sep 14 11:16:32.159159 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:32.159118 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:32.159539 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:32.159370 2556 patch_prober.go:28] interesting pod/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Sep 14 11:16:32.159539 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:32.159445 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" podUID="0376711b-9ae6-441c-8083-bf6086d85862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Sep 14 11:16:33.159220 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:33.159184 2556 patch_prober.go:28] interesting pod/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Sep 14 11:16:33.159699 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:33.159243 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" podUID="0376711b-9ae6-441c-8083-bf6086d85862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Sep 14 11:16:34.158797 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:34.158762 2556 patch_prober.go:28] interesting pod/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Sep 14 11:16:34.158941 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:34.158815 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" podUID="0376711b-9ae6-441c-8083-bf6086d85862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Sep 14 11:16:34.247105 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:34.247063 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz" event={"ID":"6b091ec2-5232-4776-8fdd-4029894256a1","Type":"ContainerStarted","Data":"dd22b4709747bce60cc2d952f72adb37418adc65b947e02563196f705f8d0939"} Sep 14 11:16:34.267699 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:34.267653 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz" podStartSLOduration=1.524631114 podStartE2EDuration="4.267639028s" totalImagesPullingTime="2.743007914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:16:34.266316424 +0000 UTC m=+952.329719334" watchObservedRunningTime="2026-09-14 11:16:34.267639028 +0000 UTC m=+952.331041938" Sep 14 11:16:35.158539 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:35.158495 2556 patch_prober.go:28] interesting pod/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Sep 14 11:16:35.158735 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:35.158562 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" podUID="0376711b-9ae6-441c-8083-bf6086d85862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Sep 14 11:16:35.252030 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:35.251997 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz" Sep 14 11:16:36.158568 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:36.158534 2556 patch_prober.go:28] interesting pod/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Sep 14 11:16:36.158740 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:36.158606 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" podUID="0376711b-9ae6-441c-8083-bf6086d85862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Sep 14 11:16:37.162849 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:37.162816 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:37.163248 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:37.163128 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:37.163867 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:37.163847 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:16:41.262301 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:41.262267 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz" Sep 14 11:16:49.331548 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:49.331464 2556 generic.go:356] "Generic (PLEG): container finished" podID="a034a7a3-4e1d-479f-bc00-14c64bc82031" containerID="3e1f4c28e50c6130e0a5542168adefe62d00080476d8b6062b444b18358df40c" exitCode=0 Sep 14 11:16:49.331912 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:49.331539 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" event={"ID":"a034a7a3-4e1d-479f-bc00-14c64bc82031","Type":"ContainerDied","Data":"3e1f4c28e50c6130e0a5542168adefe62d00080476d8b6062b444b18358df40c"} Sep 14 11:16:49.331952 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:49.331927 2556 scope.go:117] "RemoveContainer" containerID="3e1f4c28e50c6130e0a5542168adefe62d00080476d8b6062b444b18358df40c" Sep 14 11:16:49.333053 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:49.333024 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:16:49.333334 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:49.333106 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:16:49.333334 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:49.333150 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:16:50.338260 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:50.338230 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" event={"ID":"a034a7a3-4e1d-479f-bc00-14c64bc82031","Type":"ContainerStarted","Data":"400488be6fd7cadbbe60bb1667f4b00e3912a5c427b1174f387b37452a990068"} Sep 14 11:16:50.739455 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:50.739425 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:50.740616 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:50.740591 2556 patch_prober.go:28] interesting pod/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Sep 14 11:16:50.740726 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:50.740638 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" podUID="a034a7a3-4e1d-479f-bc00-14c64bc82031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Sep 14 11:16:51.739210 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:51.739177 2556 patch_prober.go:28] interesting pod/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Sep 14 11:16:51.739600 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:51.739230 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" podUID="a034a7a3-4e1d-479f-bc00-14c64bc82031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Sep 14 11:16:52.739589 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:52.739555 2556 patch_prober.go:28] interesting pod/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Sep 14 11:16:52.740050 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:52.739612 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" podUID="a034a7a3-4e1d-479f-bc00-14c64bc82031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Sep 14 11:16:53.739506 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:53.739468 2556 patch_prober.go:28] interesting pod/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Sep 14 11:16:53.739686 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:53.739528 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" podUID="a034a7a3-4e1d-479f-bc00-14c64bc82031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Sep 14 11:16:54.738937 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:54.738896 2556 patch_prober.go:28] interesting pod/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Sep 14 11:16:54.739133 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:54.738967 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" podUID="a034a7a3-4e1d-479f-bc00-14c64bc82031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Sep 14 11:16:55.184841 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:55.184800 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:55.327031 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:55.326991 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ac71380-pr8tj"] Sep 14 11:16:55.329405 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:55.329387 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ac71380-pr8tj" Sep 14 11:16:55.332389 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:55.332368 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f98vt\"" Sep 14 11:16:55.342200 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:55.342173 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ac71380-pr8tj"] Sep 14 11:16:55.345363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:55.345338 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d397af2-81dc-47e3-b06e-dd57c1fbc794-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1ac71380-pr8tj\" (UID: \"9d397af2-81dc-47e3-b06e-dd57c1fbc794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ac71380-pr8tj" Sep 14 11:16:55.446424 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:55.446341 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d397af2-81dc-47e3-b06e-dd57c1fbc794-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1ac71380-pr8tj\" (UID: \"9d397af2-81dc-47e3-b06e-dd57c1fbc794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ac71380-pr8tj" Sep 14 11:16:55.446933 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:55.446911 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d397af2-81dc-47e3-b06e-dd57c1fbc794-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1ac71380-pr8tj\" (UID: \"9d397af2-81dc-47e3-b06e-dd57c1fbc794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ac71380-pr8tj" Sep 14 11:16:55.639577 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:55.639549 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ac71380-pr8tj" Sep 14 11:16:55.743394 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:55.743324 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:55.744577 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:55.744554 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:16:55.912506 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:55.912483 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ac71380-pr8tj"] Sep 14 11:16:55.915079 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:16:55.915049 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d397af2_81dc_47e3_b06e_dd57c1fbc794.slice/crio-dc5037289f06bb0deb1ca1a7a38f2d9641d469716228031801d335c20da12861 WatchSource:0}: Error finding container dc5037289f06bb0deb1ca1a7a38f2d9641d469716228031801d335c20da12861: Status 404 returned error can't find the container with id dc5037289f06bb0deb1ca1a7a38f2d9641d469716228031801d335c20da12861 Sep 14 11:16:56.370983 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:56.370956 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ac71380-pr8tj" event={"ID":"9d397af2-81dc-47e3-b06e-dd57c1fbc794","Type":"ContainerStarted","Data":"2565ad8f2ebaa96ee725699dcfb3467b9695dfa757694ce9a1320da030f13a72"} Sep 14 11:16:56.371317 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:56.371006 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ac71380-pr8tj" event={"ID":"9d397af2-81dc-47e3-b06e-dd57c1fbc794","Type":"ContainerStarted","Data":"dc5037289f06bb0deb1ca1a7a38f2d9641d469716228031801d335c20da12861"} Sep 14 11:16:56.395959 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:56.395909 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ac71380-pr8tj" podStartSLOduration=1.395892394 podStartE2EDuration="1.395892394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:16:56.394705098 +0000 UTC m=+974.458108009" watchObservedRunningTime="2026-09-14 11:16:56.395892394 +0000 UTC m=+974.459295332" Sep 14 11:16:57.377467 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:57.377431 2556 generic.go:356] "Generic (PLEG): container finished" podID="9d397af2-81dc-47e3-b06e-dd57c1fbc794" containerID="2565ad8f2ebaa96ee725699dcfb3467b9695dfa757694ce9a1320da030f13a72" exitCode=0 Sep 14 11:16:57.377826 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:57.377478 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ac71380-pr8tj" event={"ID":"9d397af2-81dc-47e3-b06e-dd57c1fbc794","Type":"ContainerDied","Data":"2565ad8f2ebaa96ee725699dcfb3467b9695dfa757694ce9a1320da030f13a72"} Sep 14 11:16:58.512986 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:58.512965 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ac71380-pr8tj" Sep 14 11:16:58.575913 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:58.575888 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d397af2-81dc-47e3-b06e-dd57c1fbc794-maas-api-service-ca\") pod \"9d397af2-81dc-47e3-b06e-dd57c1fbc794\" (UID: \"9d397af2-81dc-47e3-b06e-dd57c1fbc794\") " Sep 14 11:16:58.576239 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:58.576215 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d397af2-81dc-47e3-b06e-dd57c1fbc794-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9d397af2-81dc-47e3-b06e-dd57c1fbc794" (UID: "9d397af2-81dc-47e3-b06e-dd57c1fbc794"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:16:58.677145 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:58.677068 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d397af2-81dc-47e3-b06e-dd57c1fbc794-maas-api-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:16:59.390693 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:59.390665 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ac71380-pr8tj" Sep 14 11:16:59.390849 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:59.390664 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ac71380-pr8tj" event={"ID":"9d397af2-81dc-47e3-b06e-dd57c1fbc794","Type":"ContainerDied","Data":"dc5037289f06bb0deb1ca1a7a38f2d9641d469716228031801d335c20da12861"} Sep 14 11:16:59.390849 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:16:59.390795 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc5037289f06bb0deb1ca1a7a38f2d9641d469716228031801d335c20da12861" Sep 14 11:17:07.013159 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.013065 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz"] Sep 14 11:17:07.013572 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.013291 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz" podUID="6b091ec2-5232-4776-8fdd-4029894256a1" containerName="maas-api" containerID="cri-o://dd22b4709747bce60cc2d952f72adb37418adc65b947e02563196f705f8d0939" gracePeriod=30 Sep 14 11:17:07.250235 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.250212 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz" Sep 14 11:17:07.342105 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.342005 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ac71380-pr8tj"] Sep 14 11:17:07.346746 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.346718 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ac71380-pr8tj"] Sep 14 11:17:07.355400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.355379 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b091ec2-5232-4776-8fdd-4029894256a1-openshift-service-ca\") pod \"6b091ec2-5232-4776-8fdd-4029894256a1\" (UID: \"6b091ec2-5232-4776-8fdd-4029894256a1\") " Sep 14 11:17:07.355509 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.355462 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b091ec2-5232-4776-8fdd-4029894256a1-metrics-certs\") pod \"6b091ec2-5232-4776-8fdd-4029894256a1\" (UID: \"6b091ec2-5232-4776-8fdd-4029894256a1\") " Sep 14 11:17:07.355509 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.355498 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b091ec2-5232-4776-8fdd-4029894256a1-maas-api-tls\") pod \"6b091ec2-5232-4776-8fdd-4029894256a1\" (UID: \"6b091ec2-5232-4776-8fdd-4029894256a1\") " Sep 14 11:17:07.355597 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.355520 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4f2f\" (UniqueName: \"kubernetes.io/projected/6b091ec2-5232-4776-8fdd-4029894256a1-kube-api-access-j4f2f\") pod \"6b091ec2-5232-4776-8fdd-4029894256a1\" (UID: \"6b091ec2-5232-4776-8fdd-4029894256a1\") " Sep 14 11:17:07.355755 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.355730 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b091ec2-5232-4776-8fdd-4029894256a1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6b091ec2-5232-4776-8fdd-4029894256a1" (UID: "6b091ec2-5232-4776-8fdd-4029894256a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:17:07.357756 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.357720 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b091ec2-5232-4776-8fdd-4029894256a1-kube-api-access-j4f2f" (OuterVolumeSpecName: "kube-api-access-j4f2f") pod "6b091ec2-5232-4776-8fdd-4029894256a1" (UID: "6b091ec2-5232-4776-8fdd-4029894256a1"). InnerVolumeSpecName "kube-api-access-j4f2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:17:07.357756 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.357742 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b091ec2-5232-4776-8fdd-4029894256a1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6b091ec2-5232-4776-8fdd-4029894256a1" (UID: "6b091ec2-5232-4776-8fdd-4029894256a1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:17:07.357756 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.357752 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b091ec2-5232-4776-8fdd-4029894256a1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6b091ec2-5232-4776-8fdd-4029894256a1" (UID: "6b091ec2-5232-4776-8fdd-4029894256a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:17:07.429613 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.429578 2556 generic.go:356] "Generic (PLEG): container finished" podID="6b091ec2-5232-4776-8fdd-4029894256a1" containerID="dd22b4709747bce60cc2d952f72adb37418adc65b947e02563196f705f8d0939" exitCode=0 Sep 14 11:17:07.429751 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.429627 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz" Sep 14 11:17:07.429751 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.429666 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz" event={"ID":"6b091ec2-5232-4776-8fdd-4029894256a1","Type":"ContainerDied","Data":"dd22b4709747bce60cc2d952f72adb37418adc65b947e02563196f705f8d0939"} Sep 14 11:17:07.429751 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.429719 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz" event={"ID":"6b091ec2-5232-4776-8fdd-4029894256a1","Type":"ContainerDied","Data":"ddf0edfb8832aa6c4638355e8191aa7202ac48d12b82e22eb3123b7b23fce72a"} Sep 14 11:17:07.429751 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.429740 2556 scope.go:117] "RemoveContainer" containerID="dd22b4709747bce60cc2d952f72adb37418adc65b947e02563196f705f8d0939" Sep 14 11:17:07.438843 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.438824 2556 scope.go:117] "RemoveContainer" containerID="dd22b4709747bce60cc2d952f72adb37418adc65b947e02563196f705f8d0939" Sep 14 11:17:07.439129 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:17:07.439080 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd22b4709747bce60cc2d952f72adb37418adc65b947e02563196f705f8d0939\": container with ID starting with dd22b4709747bce60cc2d952f72adb37418adc65b947e02563196f705f8d0939 not found: ID does not exist" containerID="dd22b4709747bce60cc2d952f72adb37418adc65b947e02563196f705f8d0939" Sep 14 11:17:07.439207 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.439141 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd22b4709747bce60cc2d952f72adb37418adc65b947e02563196f705f8d0939"} err="failed to get container status \"dd22b4709747bce60cc2d952f72adb37418adc65b947e02563196f705f8d0939\": rpc error: code = NotFound desc = could not find container \"dd22b4709747bce60cc2d952f72adb37418adc65b947e02563196f705f8d0939\": container with ID starting with dd22b4709747bce60cc2d952f72adb37418adc65b947e02563196f705f8d0939 not found: ID does not exist" Sep 14 11:17:07.455190 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.455160 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz"] Sep 14 11:17:07.456601 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.456573 2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b091ec2-5232-4776-8fdd-4029894256a1-metrics-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:17:07.456601 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.456592 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b091ec2-5232-4776-8fdd-4029894256a1-maas-api-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:17:07.456601 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.456600 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4f2f\" (UniqueName: \"kubernetes.io/projected/6b091ec2-5232-4776-8fdd-4029894256a1-kube-api-access-j4f2f\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:17:07.456794 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.456609 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b091ec2-5232-4776-8fdd-4029894256a1-openshift-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:17:07.467795 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.467770 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ac71380-579c4d96f9-c6jxz"] Sep 14 11:17:07.488794 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.488772 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j"] Sep 14 11:17:07.489073 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:07.489044 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" podUID="a034a7a3-4e1d-479f-bc00-14c64bc82031" containerName="istio-proxy" containerID="cri-o://400488be6fd7cadbbe60bb1667f4b00e3912a5c427b1174f387b37452a990068" gracePeriod=30 Sep 14 11:17:08.382716 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.382678 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b091ec2-5232-4776-8fdd-4029894256a1" path="/var/lib/kubelet/pods/6b091ec2-5232-4776-8fdd-4029894256a1/volumes" Sep 14 11:17:08.383224 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.383204 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d397af2-81dc-47e3-b06e-dd57c1fbc794" path="/var/lib/kubelet/pods/9d397af2-81dc-47e3-b06e-dd57c1fbc794/volumes" Sep 14 11:17:08.555010 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.554981 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r"] Sep 14 11:17:08.555374 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.555361 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d397af2-81dc-47e3-b06e-dd57c1fbc794" containerName="revoke-keys" Sep 14 11:17:08.555428 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.555376 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d397af2-81dc-47e3-b06e-dd57c1fbc794" containerName="revoke-keys" Sep 14 11:17:08.555428 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.555394 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b091ec2-5232-4776-8fdd-4029894256a1" containerName="maas-api" Sep 14 11:17:08.555428 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.555400 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b091ec2-5232-4776-8fdd-4029894256a1" containerName="maas-api" Sep 14 11:17:08.555521 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.555449 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b091ec2-5232-4776-8fdd-4029894256a1" containerName="maas-api" Sep 14 11:17:08.555521 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.555455 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d397af2-81dc-47e3-b06e-dd57c1fbc794" containerName="revoke-keys" Sep 14 11:17:08.558098 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.558066 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.563237 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.563207 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-c652e0cd-gw-openshift-default-dockercfg-n59z8\"" Sep 14 11:17:08.580779 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.580755 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r"] Sep 14 11:17:08.667757 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.667656 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c975f8-8c02-4961-8567-c7408c61262b-istio-podinfo\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.667757 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.667723 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-istio-data\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.668003 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.667766 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-istio-envoy\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.668003 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.667791 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-credential-socket\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.668003 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.667854 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j897\" (UniqueName: \"kubernetes.io/projected/09c975f8-8c02-4961-8567-c7408c61262b-kube-api-access-6j897\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.668003 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.667952 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-workload-certs\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.668003 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.667982 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c975f8-8c02-4961-8567-c7408c61262b-istio-token\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.668272 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.668006 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c975f8-8c02-4961-8567-c7408c61262b-istiod-ca-cert\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.668272 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.668075 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-workload-socket\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.769029 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.768999 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-istio-data\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.769233 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.769040 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-istio-envoy\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.769233 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.769061 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-credential-socket\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.769233 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.769130 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j897\" (UniqueName: \"kubernetes.io/projected/09c975f8-8c02-4961-8567-c7408c61262b-kube-api-access-6j897\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.769233 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.769187 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-workload-certs\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.769233 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.769205 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c975f8-8c02-4961-8567-c7408c61262b-istio-token\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.769233 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.769220 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c975f8-8c02-4961-8567-c7408c61262b-istiod-ca-cert\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.769530 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.769274 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-workload-socket\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.769530 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.769304 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c975f8-8c02-4961-8567-c7408c61262b-istio-podinfo\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.769634 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.769605 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-credential-socket\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.769802 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.769777 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-workload-socket\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.769929 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.769797 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-workload-certs\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.770024 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.770001 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-istio-data\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.770146 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.770126 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c975f8-8c02-4961-8567-c7408c61262b-istiod-ca-cert\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.771629 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.771612 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c975f8-8c02-4961-8567-c7408c61262b-istio-podinfo\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.771822 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.771801 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-istio-envoy\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.789415 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.789383 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c975f8-8c02-4961-8567-c7408c61262b-istio-token\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.790176 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.790148 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j897\" (UniqueName: \"kubernetes.io/projected/09c975f8-8c02-4961-8567-c7408c61262b-kube-api-access-6j897\") pod \"e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:08.874352 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:08.874309 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:09.182655 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:09.182622 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r"] Sep 14 11:17:09.184164 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:17:09.184132 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c975f8_8c02_4961_8567_c7408c61262b.slice/crio-696488f9a0090b71e2a658c98cab631a6ddec38334cab89414bdc624ae4a1713 WatchSource:0}: Error finding container 696488f9a0090b71e2a658c98cab631a6ddec38334cab89414bdc624ae4a1713: Status 404 returned error can't find the container with id 696488f9a0090b71e2a658c98cab631a6ddec38334cab89414bdc624ae4a1713 Sep 14 11:17:09.186490 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:09.186451 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:17:09.186601 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:09.186535 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:17:09.186601 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:09.186578 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:17:09.445551 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:09.445514 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" event={"ID":"09c975f8-8c02-4961-8567-c7408c61262b","Type":"ContainerStarted","Data":"4744ac8681cf44424162fdbf1f9a93b8a78ac820b07aa0c6a34093b62de533cb"} Sep 14 11:17:09.446080 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:09.446061 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" event={"ID":"09c975f8-8c02-4961-8567-c7408c61262b","Type":"ContainerStarted","Data":"696488f9a0090b71e2a658c98cab631a6ddec38334cab89414bdc624ae4a1713"} Sep 14 11:17:09.498846 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:09.498738 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" podStartSLOduration=1.498716872 podStartE2EDuration="1.498716872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:17:09.497353814 +0000 UTC m=+987.560756731" watchObservedRunningTime="2026-09-14 11:17:09.498716872 +0000 UTC m=+987.562119781" Sep 14 11:17:09.875133 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:09.875076 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:10.745993 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:10.745955 2556 patch_prober.go:28] interesting pod/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 14 11:17:10.746379 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:10.746011 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" podUID="a034a7a3-4e1d-479f-bc00-14c64bc82031" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 14 11:17:10.880185 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:10.880155 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:11.456937 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:11.456908 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:11.457849 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:11.457828 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:12.731762 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.731738 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:17:12.805822 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.805745 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcz42\" (UniqueName: \"kubernetes.io/projected/a034a7a3-4e1d-479f-bc00-14c64bc82031-kube-api-access-qcz42\") pod \"a034a7a3-4e1d-479f-bc00-14c64bc82031\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " Sep 14 11:17:12.805822 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.805779 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-istio-envoy\") pod \"a034a7a3-4e1d-479f-bc00-14c64bc82031\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " Sep 14 11:17:12.805822 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.805813 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a034a7a3-4e1d-479f-bc00-14c64bc82031-istio-token\") pod \"a034a7a3-4e1d-479f-bc00-14c64bc82031\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " Sep 14 11:17:12.806112 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.805840 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-workload-certs\") pod \"a034a7a3-4e1d-479f-bc00-14c64bc82031\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " Sep 14 11:17:12.806112 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.805953 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-credential-socket\") pod \"a034a7a3-4e1d-479f-bc00-14c64bc82031\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " Sep 14 11:17:12.806112 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.805991 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-istio-data\") pod \"a034a7a3-4e1d-479f-bc00-14c64bc82031\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " Sep 14 11:17:12.806112 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.806069 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a034a7a3-4e1d-479f-bc00-14c64bc82031-istiod-ca-cert\") pod \"a034a7a3-4e1d-479f-bc00-14c64bc82031\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " Sep 14 11:17:12.806112 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.806113 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a034a7a3-4e1d-479f-bc00-14c64bc82031-istio-podinfo\") pod \"a034a7a3-4e1d-479f-bc00-14c64bc82031\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " Sep 14 11:17:12.806383 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.806128 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a034a7a3-4e1d-479f-bc00-14c64bc82031" (UID: "a034a7a3-4e1d-479f-bc00-14c64bc82031"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:17:12.806383 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.806201 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-workload-socket\") pod \"a034a7a3-4e1d-479f-bc00-14c64bc82031\" (UID: \"a034a7a3-4e1d-479f-bc00-14c64bc82031\") " Sep 14 11:17:12.806383 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.806234 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a034a7a3-4e1d-479f-bc00-14c64bc82031" (UID: "a034a7a3-4e1d-479f-bc00-14c64bc82031"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:17:12.806383 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.806286 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-istio-data" (OuterVolumeSpecName: "istio-data") pod "a034a7a3-4e1d-479f-bc00-14c64bc82031" (UID: "a034a7a3-4e1d-479f-bc00-14c64bc82031"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:17:12.806585 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.806560 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a034a7a3-4e1d-479f-bc00-14c64bc82031-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a034a7a3-4e1d-479f-bc00-14c64bc82031" (UID: "a034a7a3-4e1d-479f-bc00-14c64bc82031"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:17:12.806635 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.806615 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:17:12.806686 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.806639 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:17:12.806686 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.806653 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:17:12.806686 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.806664 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a034a7a3-4e1d-479f-bc00-14c64bc82031" (UID: "a034a7a3-4e1d-479f-bc00-14c64bc82031"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:17:12.808199 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.808172 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a034a7a3-4e1d-479f-bc00-14c64bc82031-istio-token" (OuterVolumeSpecName: "istio-token") pod "a034a7a3-4e1d-479f-bc00-14c64bc82031" (UID: "a034a7a3-4e1d-479f-bc00-14c64bc82031"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:17:12.808199 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.808187 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a034a7a3-4e1d-479f-bc00-14c64bc82031-kube-api-access-qcz42" (OuterVolumeSpecName: "kube-api-access-qcz42") pod "a034a7a3-4e1d-479f-bc00-14c64bc82031" (UID: "a034a7a3-4e1d-479f-bc00-14c64bc82031"). InnerVolumeSpecName "kube-api-access-qcz42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:17:12.808377 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.808296 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a034a7a3-4e1d-479f-bc00-14c64bc82031" (UID: "a034a7a3-4e1d-479f-bc00-14c64bc82031"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:17:12.808433 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.808403 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a034a7a3-4e1d-479f-bc00-14c64bc82031-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a034a7a3-4e1d-479f-bc00-14c64bc82031" (UID: "a034a7a3-4e1d-479f-bc00-14c64bc82031"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 11:17:12.907362 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.907326 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a034a7a3-4e1d-479f-bc00-14c64bc82031-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:17:12.907362 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.907355 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a034a7a3-4e1d-479f-bc00-14c64bc82031-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:17:12.907362 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.907368 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:17:12.907584 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.907380 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcz42\" (UniqueName: \"kubernetes.io/projected/a034a7a3-4e1d-479f-bc00-14c64bc82031-kube-api-access-qcz42\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:17:12.907584 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.907392 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a034a7a3-4e1d-479f-bc00-14c64bc82031-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:17:12.907584 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:12.907402 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a034a7a3-4e1d-479f-bc00-14c64bc82031-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:17:13.466552 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:13.466518 2556 generic.go:356] "Generic (PLEG): container finished" podID="a034a7a3-4e1d-479f-bc00-14c64bc82031" containerID="400488be6fd7cadbbe60bb1667f4b00e3912a5c427b1174f387b37452a990068" exitCode=0 Sep 14 11:17:13.466708 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:13.466567 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" Sep 14 11:17:13.466708 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:13.466600 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" event={"ID":"a034a7a3-4e1d-479f-bc00-14c64bc82031","Type":"ContainerDied","Data":"400488be6fd7cadbbe60bb1667f4b00e3912a5c427b1174f387b37452a990068"} Sep 14 11:17:13.466708 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:13.466649 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j" event={"ID":"a034a7a3-4e1d-479f-bc00-14c64bc82031","Type":"ContainerDied","Data":"006dacd1f7e8d40f2de29900fa67e7101dd2624072c8a6fa0c24f1f9d4df53d0"} Sep 14 11:17:13.466708 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:13.466671 2556 scope.go:117] "RemoveContainer" containerID="400488be6fd7cadbbe60bb1667f4b00e3912a5c427b1174f387b37452a990068" Sep 14 11:17:13.476305 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:13.476290 2556 scope.go:117] "RemoveContainer" containerID="3e1f4c28e50c6130e0a5542168adefe62d00080476d8b6062b444b18358df40c" Sep 14 11:17:13.484142 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:13.484123 2556 scope.go:117] "RemoveContainer" containerID="400488be6fd7cadbbe60bb1667f4b00e3912a5c427b1174f387b37452a990068" Sep 14 11:17:13.484400 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:17:13.484383 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"400488be6fd7cadbbe60bb1667f4b00e3912a5c427b1174f387b37452a990068\": container with ID starting with 400488be6fd7cadbbe60bb1667f4b00e3912a5c427b1174f387b37452a990068 not found: ID does not exist" containerID="400488be6fd7cadbbe60bb1667f4b00e3912a5c427b1174f387b37452a990068" Sep 14 11:17:13.484476 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:13.484407 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"400488be6fd7cadbbe60bb1667f4b00e3912a5c427b1174f387b37452a990068"} err="failed to get container status \"400488be6fd7cadbbe60bb1667f4b00e3912a5c427b1174f387b37452a990068\": rpc error: code = NotFound desc = could not find container \"400488be6fd7cadbbe60bb1667f4b00e3912a5c427b1174f387b37452a990068\": container with ID starting with 400488be6fd7cadbbe60bb1667f4b00e3912a5c427b1174f387b37452a990068 not found: ID does not exist" Sep 14 11:17:13.484476 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:13.484426 2556 scope.go:117] "RemoveContainer" containerID="3e1f4c28e50c6130e0a5542168adefe62d00080476d8b6062b444b18358df40c" Sep 14 11:17:13.484694 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:17:13.484677 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e1f4c28e50c6130e0a5542168adefe62d00080476d8b6062b444b18358df40c\": container with ID starting with 3e1f4c28e50c6130e0a5542168adefe62d00080476d8b6062b444b18358df40c not found: ID does not exist" containerID="3e1f4c28e50c6130e0a5542168adefe62d00080476d8b6062b444b18358df40c" Sep 14 11:17:13.484746 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:13.484698 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e1f4c28e50c6130e0a5542168adefe62d00080476d8b6062b444b18358df40c"} err="failed to get container status \"3e1f4c28e50c6130e0a5542168adefe62d00080476d8b6062b444b18358df40c\": rpc error: code = NotFound desc = could not find container \"3e1f4c28e50c6130e0a5542168adefe62d00080476d8b6062b444b18358df40c\": container with ID starting with 3e1f4c28e50c6130e0a5542168adefe62d00080476d8b6062b444b18358df40c not found: ID does not exist" Sep 14 11:17:13.491762 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:13.491741 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j"] Sep 14 11:17:13.493413 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:13.493391 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1ac71380-openshift-default-55d679d984-j4c7j"] Sep 14 11:17:14.173261 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.173228 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8"] Sep 14 11:17:14.173728 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.173712 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a034a7a3-4e1d-479f-bc00-14c64bc82031" containerName="istio-proxy" Sep 14 11:17:14.173774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.173731 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a034a7a3-4e1d-479f-bc00-14c64bc82031" containerName="istio-proxy" Sep 14 11:17:14.173774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.173760 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a034a7a3-4e1d-479f-bc00-14c64bc82031" containerName="istio-proxy" Sep 14 11:17:14.173774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.173769 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a034a7a3-4e1d-479f-bc00-14c64bc82031" containerName="istio-proxy" Sep 14 11:17:14.173872 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.173848 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a034a7a3-4e1d-479f-bc00-14c64bc82031" containerName="istio-proxy" Sep 14 11:17:14.173872 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.173860 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a034a7a3-4e1d-479f-bc00-14c64bc82031" containerName="istio-proxy" Sep 14 11:17:14.178493 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.178469 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" Sep 14 11:17:14.183708 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.183605 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-c652e0cd-serving-cert\"" Sep 14 11:17:14.183708 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.183686 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 14 11:17:14.184017 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.183902 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 14 11:17:14.185185 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.184920 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 11:17:14.185287 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.185243 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w7q69\"" Sep 14 11:17:14.189488 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.189212 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8"] Sep 14 11:17:14.321457 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.321423 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-openshift-service-ca\") pod \"maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8\" (UID: \"3538e2e2-733f-4a08-b917-9a11a4f5a4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" Sep 14 11:17:14.321620 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.321471 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-metrics-certs\") pod \"maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8\" (UID: \"3538e2e2-733f-4a08-b917-9a11a4f5a4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" Sep 14 11:17:14.321620 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.321505 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v55lv\" (UniqueName: \"kubernetes.io/projected/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-kube-api-access-v55lv\") pod \"maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8\" (UID: \"3538e2e2-733f-4a08-b917-9a11a4f5a4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" Sep 14 11:17:14.321620 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.321525 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-maas-api-tls\") pod \"maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8\" (UID: \"3538e2e2-733f-4a08-b917-9a11a4f5a4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" Sep 14 11:17:14.382297 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.382265 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a034a7a3-4e1d-479f-bc00-14c64bc82031" path="/var/lib/kubelet/pods/a034a7a3-4e1d-479f-bc00-14c64bc82031/volumes" Sep 14 11:17:14.422723 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.422693 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-openshift-service-ca\") pod \"maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8\" (UID: \"3538e2e2-733f-4a08-b917-9a11a4f5a4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" Sep 14 11:17:14.422842 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.422732 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-metrics-certs\") pod \"maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8\" (UID: \"3538e2e2-733f-4a08-b917-9a11a4f5a4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" Sep 14 11:17:14.422842 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.422832 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v55lv\" (UniqueName: \"kubernetes.io/projected/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-kube-api-access-v55lv\") pod \"maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8\" (UID: \"3538e2e2-733f-4a08-b917-9a11a4f5a4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" Sep 14 11:17:14.422930 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.422859 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-maas-api-tls\") pod \"maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8\" (UID: \"3538e2e2-733f-4a08-b917-9a11a4f5a4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" Sep 14 11:17:14.423001 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:17:14.422986 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-serving-cert: secret "maas-api-e2e-migrate-c652e0cd-serving-cert" not found Sep 14 11:17:14.423066 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:17:14.423056 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-maas-api-tls podName:3538e2e2-733f-4a08-b917-9a11a4f5a4fa nodeName:}" failed. No retries permitted until 2026-09-14 11:17:14.923037237 +0000 UTC m=+992.986440130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-maas-api-tls") pod "maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" (UID: "3538e2e2-733f-4a08-b917-9a11a4f5a4fa") : secret "maas-api-e2e-migrate-c652e0cd-serving-cert" not found Sep 14 11:17:14.423402 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.423344 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-openshift-service-ca\") pod \"maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8\" (UID: \"3538e2e2-733f-4a08-b917-9a11a4f5a4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" Sep 14 11:17:14.425128 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.425103 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-metrics-certs\") pod \"maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8\" (UID: \"3538e2e2-733f-4a08-b917-9a11a4f5a4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" Sep 14 11:17:14.431881 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.431859 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v55lv\" (UniqueName: \"kubernetes.io/projected/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-kube-api-access-v55lv\") pod \"maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8\" (UID: \"3538e2e2-733f-4a08-b917-9a11a4f5a4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" Sep 14 11:17:14.929265 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.929236 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-maas-api-tls\") pod \"maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8\" (UID: \"3538e2e2-733f-4a08-b917-9a11a4f5a4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" Sep 14 11:17:14.931772 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:14.931739 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-maas-api-tls\") pod \"maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8\" (UID: \"3538e2e2-733f-4a08-b917-9a11a4f5a4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" Sep 14 11:17:15.094338 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:15.094299 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" Sep 14 11:17:15.370439 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:15.370416 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8"] Sep 14 11:17:15.372816 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:17:15.372791 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3538e2e2_733f_4a08_b917_9a11a4f5a4fa.slice/crio-ace83d6c179b007de8973d72d921ca534bbb91f67bf699086537bde8b1ffaa5b WatchSource:0}: Error finding container ace83d6c179b007de8973d72d921ca534bbb91f67bf699086537bde8b1ffaa5b: Status 404 returned error can't find the container with id ace83d6c179b007de8973d72d921ca534bbb91f67bf699086537bde8b1ffaa5b Sep 14 11:17:15.479217 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:15.479181 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" event={"ID":"3538e2e2-733f-4a08-b917-9a11a4f5a4fa","Type":"ContainerStarted","Data":"ace83d6c179b007de8973d72d921ca534bbb91f67bf699086537bde8b1ffaa5b"} Sep 14 11:17:16.487124 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:16.487071 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" event={"ID":"3538e2e2-733f-4a08-b917-9a11a4f5a4fa","Type":"ContainerStarted","Data":"825a602074663cf627493a59f4fa922f3aa6dd7c1b1365c52f0d3b94ad0dc1ab"} Sep 14 11:17:16.487577 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:16.487161 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" Sep 14 11:17:22.497593 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:22.497561 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" Sep 14 11:17:22.536776 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:22.536730 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" podStartSLOduration=8.183216136 podStartE2EDuration="8.536716065s" totalImagesPullingTime="353.499929ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:17:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:17:16.52020197 +0000 UTC m=+994.583604879" watchObservedRunningTime="2026-09-14 11:17:22.536716065 +0000 UTC m=+1000.600118974" Sep 14 11:17:33.570361 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:33.570330 2556 generic.go:356] "Generic (PLEG): container finished" podID="09c975f8-8c02-4961-8567-c7408c61262b" containerID="4744ac8681cf44424162fdbf1f9a93b8a78ac820b07aa0c6a34093b62de533cb" exitCode=0 Sep 14 11:17:33.570867 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:33.570365 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" event={"ID":"09c975f8-8c02-4961-8567-c7408c61262b","Type":"ContainerDied","Data":"4744ac8681cf44424162fdbf1f9a93b8a78ac820b07aa0c6a34093b62de533cb"} Sep 14 11:17:33.570943 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:33.570865 2556 scope.go:117] "RemoveContainer" containerID="4744ac8681cf44424162fdbf1f9a93b8a78ac820b07aa0c6a34093b62de533cb" Sep 14 11:17:33.572043 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:33.572016 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:17:33.572120 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:33.572100 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:17:33.572162 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:33.572132 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:17:34.587448 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:34.587405 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" event={"ID":"09c975f8-8c02-4961-8567-c7408c61262b","Type":"ContainerStarted","Data":"4c85a8d5c7eb4877ec86b4b428056022fd09202c9a6d76544538c58a18e9cd99"} Sep 14 11:17:34.874759 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:34.874735 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:34.874997 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:34.874974 2556 patch_prober.go:28] interesting pod/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 14 11:17:34.875056 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:34.875034 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" podUID="09c975f8-8c02-4961-8567-c7408c61262b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 14 11:17:35.874897 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:35.874864 2556 patch_prober.go:28] interesting pod/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 14 11:17:35.875288 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:35.874923 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" podUID="09c975f8-8c02-4961-8567-c7408c61262b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 14 11:17:36.875108 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:36.875049 2556 patch_prober.go:28] interesting pod/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 14 11:17:36.875564 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:36.875147 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" podUID="09c975f8-8c02-4961-8567-c7408c61262b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 14 11:17:37.874951 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:37.874917 2556 patch_prober.go:28] interesting pod/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 14 11:17:37.875156 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:37.874969 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" podUID="09c975f8-8c02-4961-8567-c7408c61262b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 14 11:17:38.874906 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:38.874875 2556 patch_prober.go:28] interesting pod/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 14 11:17:38.875080 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:38.874937 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" podUID="09c975f8-8c02-4961-8567-c7408c61262b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 14 11:17:39.879556 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:39.879523 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:39.879986 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:39.879797 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:39.880567 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:39.880551 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:17:41.193258 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:41.193225 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c652e0cd-8n6t2"] Sep 14 11:17:41.195868 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:41.195851 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c652e0cd-8n6t2" Sep 14 11:17:41.199361 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:41.199336 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f98vt\"" Sep 14 11:17:41.211121 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:41.211074 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c652e0cd-8n6t2"] Sep 14 11:17:41.277888 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:41.277850 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acaf0e1c-e863-4b84-b301-70921eec6914-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c652e0cd-8n6t2\" (UID: \"acaf0e1c-e863-4b84-b301-70921eec6914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c652e0cd-8n6t2" Sep 14 11:17:41.379324 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:41.379293 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acaf0e1c-e863-4b84-b301-70921eec6914-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c652e0cd-8n6t2\" (UID: \"acaf0e1c-e863-4b84-b301-70921eec6914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c652e0cd-8n6t2" Sep 14 11:17:41.379879 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:41.379860 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acaf0e1c-e863-4b84-b301-70921eec6914-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c652e0cd-8n6t2\" (UID: \"acaf0e1c-e863-4b84-b301-70921eec6914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c652e0cd-8n6t2" Sep 14 11:17:41.506336 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:41.506265 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c652e0cd-8n6t2" Sep 14 11:17:41.792246 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:41.792221 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c652e0cd-8n6t2"] Sep 14 11:17:41.793053 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:17:41.793028 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacaf0e1c_e863_4b84_b301_70921eec6914.slice/crio-63f1f4e1cb5344394c4a9227e6bd0c00c73f7f3890ad0762f63318e7cd28cf85 WatchSource:0}: Error finding container 63f1f4e1cb5344394c4a9227e6bd0c00c73f7f3890ad0762f63318e7cd28cf85: Status 404 returned error can't find the container with id 63f1f4e1cb5344394c4a9227e6bd0c00c73f7f3890ad0762f63318e7cd28cf85 Sep 14 11:17:42.641109 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:42.641057 2556 generic.go:356] "Generic (PLEG): container finished" podID="acaf0e1c-e863-4b84-b301-70921eec6914" containerID="f73e060faad9608cf8553deaa6bdf3fb02aab411f6434c8682c1a3a079f7b441" exitCode=0 Sep 14 11:17:42.641465 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:42.641116 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c652e0cd-8n6t2" event={"ID":"acaf0e1c-e863-4b84-b301-70921eec6914","Type":"ContainerDied","Data":"f73e060faad9608cf8553deaa6bdf3fb02aab411f6434c8682c1a3a079f7b441"} Sep 14 11:17:42.641465 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:42.641208 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c652e0cd-8n6t2" event={"ID":"acaf0e1c-e863-4b84-b301-70921eec6914","Type":"ContainerStarted","Data":"63f1f4e1cb5344394c4a9227e6bd0c00c73f7f3890ad0762f63318e7cd28cf85"} Sep 14 11:17:43.765727 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:43.765704 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c652e0cd-8n6t2" Sep 14 11:17:43.802737 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:43.802712 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acaf0e1c-e863-4b84-b301-70921eec6914-maas-api-service-ca\") pod \"acaf0e1c-e863-4b84-b301-70921eec6914\" (UID: \"acaf0e1c-e863-4b84-b301-70921eec6914\") " Sep 14 11:17:43.803035 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:43.803008 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acaf0e1c-e863-4b84-b301-70921eec6914-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "acaf0e1c-e863-4b84-b301-70921eec6914" (UID: "acaf0e1c-e863-4b84-b301-70921eec6914"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:17:43.904312 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:43.904261 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acaf0e1c-e863-4b84-b301-70921eec6914-maas-api-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:17:44.651650 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:44.651564 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c652e0cd-8n6t2" Sep 14 11:17:44.651650 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:44.651570 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c652e0cd-8n6t2" event={"ID":"acaf0e1c-e863-4b84-b301-70921eec6914","Type":"ContainerDied","Data":"63f1f4e1cb5344394c4a9227e6bd0c00c73f7f3890ad0762f63318e7cd28cf85"} Sep 14 11:17:44.651650 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:44.651620 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63f1f4e1cb5344394c4a9227e6bd0c00c73f7f3890ad0762f63318e7cd28cf85" Sep 14 11:17:47.034723 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:47.034691 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-27a1a2-l92h7"] Sep 14 11:17:47.035065 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:47.035049 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acaf0e1c-e863-4b84-b301-70921eec6914" containerName="revoke-keys" Sep 14 11:17:47.035065 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:47.035060 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="acaf0e1c-e863-4b84-b301-70921eec6914" containerName="revoke-keys" Sep 14 11:17:47.035169 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:47.035135 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="acaf0e1c-e863-4b84-b301-70921eec6914" containerName="revoke-keys" Sep 14 11:17:47.037150 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:47.037134 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-27a1a2-l92h7" Sep 14 11:17:47.040195 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:47.040177 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f98vt\"" Sep 14 11:17:47.071556 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:47.071531 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-27a1a2-l92h7"] Sep 14 11:17:47.131868 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:47.131840 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf74786-a1ab-44b2-8f28-c8817abb20c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-27a1a2-l92h7\" (UID: \"9bf74786-a1ab-44b2-8f28-c8817abb20c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-27a1a2-l92h7" Sep 14 11:17:47.232305 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:47.232278 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf74786-a1ab-44b2-8f28-c8817abb20c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-27a1a2-l92h7\" (UID: \"9bf74786-a1ab-44b2-8f28-c8817abb20c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-27a1a2-l92h7" Sep 14 11:17:47.232847 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:47.232825 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf74786-a1ab-44b2-8f28-c8817abb20c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-27a1a2-l92h7\" (UID: \"9bf74786-a1ab-44b2-8f28-c8817abb20c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-27a1a2-l92h7" Sep 14 11:17:47.346261 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:47.346181 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-27a1a2-l92h7" Sep 14 11:17:47.649863 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:47.649826 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-27a1a2-l92h7"] Sep 14 11:17:47.652216 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:17:47.652192 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf74786_a1ab_44b2_8f28_c8817abb20c3.slice/crio-6b7c603547c46f0b43f02fecf64f64f20c5b3061ebbb9ce369d0b48367443801 WatchSource:0}: Error finding container 6b7c603547c46f0b43f02fecf64f64f20c5b3061ebbb9ce369d0b48367443801: Status 404 returned error can't find the container with id 6b7c603547c46f0b43f02fecf64f64f20c5b3061ebbb9ce369d0b48367443801 Sep 14 11:17:47.666906 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:47.666885 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-27a1a2-l92h7" event={"ID":"9bf74786-a1ab-44b2-8f28-c8817abb20c3","Type":"ContainerStarted","Data":"6b7c603547c46f0b43f02fecf64f64f20c5b3061ebbb9ce369d0b48367443801"} Sep 14 11:17:48.673204 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:48.673175 2556 generic.go:356] "Generic (PLEG): container finished" podID="9bf74786-a1ab-44b2-8f28-c8817abb20c3" containerID="f909b72ccbc7ead7f751cabe5245589597a338547a50e021181607c96862df54" exitCode=0 Sep 14 11:17:48.673659 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:48.673252 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-27a1a2-l92h7" event={"ID":"9bf74786-a1ab-44b2-8f28-c8817abb20c3","Type":"ContainerDied","Data":"f909b72ccbc7ead7f751cabe5245589597a338547a50e021181607c96862df54"} Sep 14 11:17:49.809604 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:49.809580 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-27a1a2-l92h7" Sep 14 11:17:49.856044 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:49.855964 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf74786-a1ab-44b2-8f28-c8817abb20c3-maas-api-service-ca\") pod \"9bf74786-a1ab-44b2-8f28-c8817abb20c3\" (UID: \"9bf74786-a1ab-44b2-8f28-c8817abb20c3\") " Sep 14 11:17:49.856503 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:49.856480 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bf74786-a1ab-44b2-8f28-c8817abb20c3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9bf74786-a1ab-44b2-8f28-c8817abb20c3" (UID: "9bf74786-a1ab-44b2-8f28-c8817abb20c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:17:49.957137 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:49.957051 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf74786-a1ab-44b2-8f28-c8817abb20c3-maas-api-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:17:50.684957 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:50.684881 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-27a1a2-l92h7" Sep 14 11:17:50.684957 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:50.684915 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-27a1a2-l92h7" event={"ID":"9bf74786-a1ab-44b2-8f28-c8817abb20c3","Type":"ContainerDied","Data":"6b7c603547c46f0b43f02fecf64f64f20c5b3061ebbb9ce369d0b48367443801"} Sep 14 11:17:50.684957 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:50.684955 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b7c603547c46f0b43f02fecf64f64f20c5b3061ebbb9ce369d0b48367443801" Sep 14 11:17:52.809000 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.808970 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp"] Sep 14 11:17:52.814583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.814541 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bf74786-a1ab-44b2-8f28-c8817abb20c3" containerName="revoke-keys" Sep 14 11:17:52.814583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.814581 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bf74786-a1ab-44b2-8f28-c8817abb20c3" containerName="revoke-keys" Sep 14 11:17:52.814877 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.814817 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bf74786-a1ab-44b2-8f28-c8817abb20c3" containerName="revoke-keys" Sep 14 11:17:52.817197 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.817178 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:17:52.821483 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.821465 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6d272c\"/\"test-model-6d272c-kserve-self-signed-certs\"" Sep 14 11:17:52.822054 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.822034 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6d272c\"/\"kube-root-ca.crt\"" Sep 14 11:17:52.822181 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.822077 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6d272c\"/\"default-dockercfg-r9dw7\"" Sep 14 11:17:52.822494 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.822481 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6d272c\"/\"openshift-service-ca.crt\"" Sep 14 11:17:52.834143 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.834082 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp"] Sep 14 11:17:52.881412 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.881376 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/994ef2fb-3939-4282-a7b3-e08247dbf4f0-model-cache\") pod \"test-model-6d272c-kserve-5589c9fc5c-r4vdp\" (UID: \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\") " pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:17:52.881532 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.881432 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/994ef2fb-3939-4282-a7b3-e08247dbf4f0-tmp-dir\") pod \"test-model-6d272c-kserve-5589c9fc5c-r4vdp\" (UID: \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\") " pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:17:52.881532 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.881503 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/994ef2fb-3939-4282-a7b3-e08247dbf4f0-tls-certs\") pod \"test-model-6d272c-kserve-5589c9fc5c-r4vdp\" (UID: \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\") " pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:17:52.881645 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.881560 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs7hj\" (UniqueName: \"kubernetes.io/projected/994ef2fb-3939-4282-a7b3-e08247dbf4f0-kube-api-access-rs7hj\") pod \"test-model-6d272c-kserve-5589c9fc5c-r4vdp\" (UID: \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\") " pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:17:52.881645 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.881606 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/994ef2fb-3939-4282-a7b3-e08247dbf4f0-dshm\") pod \"test-model-6d272c-kserve-5589c9fc5c-r4vdp\" (UID: \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\") " pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:17:52.881645 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.881623 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/994ef2fb-3939-4282-a7b3-e08247dbf4f0-home\") pod \"test-model-6d272c-kserve-5589c9fc5c-r4vdp\" (UID: \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\") " pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:17:52.982155 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.982123 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/994ef2fb-3939-4282-a7b3-e08247dbf4f0-dshm\") pod \"test-model-6d272c-kserve-5589c9fc5c-r4vdp\" (UID: \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\") " pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:17:52.982287 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.982167 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/994ef2fb-3939-4282-a7b3-e08247dbf4f0-home\") pod \"test-model-6d272c-kserve-5589c9fc5c-r4vdp\" (UID: \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\") " pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:17:52.982287 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.982198 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/994ef2fb-3939-4282-a7b3-e08247dbf4f0-model-cache\") pod \"test-model-6d272c-kserve-5589c9fc5c-r4vdp\" (UID: \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\") " pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:17:52.982392 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.982297 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/994ef2fb-3939-4282-a7b3-e08247dbf4f0-tmp-dir\") pod \"test-model-6d272c-kserve-5589c9fc5c-r4vdp\" (UID: \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\") " pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:17:52.982392 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.982366 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/994ef2fb-3939-4282-a7b3-e08247dbf4f0-tls-certs\") pod \"test-model-6d272c-kserve-5589c9fc5c-r4vdp\" (UID: \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\") " pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:17:52.982495 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.982430 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs7hj\" (UniqueName: \"kubernetes.io/projected/994ef2fb-3939-4282-a7b3-e08247dbf4f0-kube-api-access-rs7hj\") pod \"test-model-6d272c-kserve-5589c9fc5c-r4vdp\" (UID: \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\") " pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:17:52.982620 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.982598 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/994ef2fb-3939-4282-a7b3-e08247dbf4f0-model-cache\") pod \"test-model-6d272c-kserve-5589c9fc5c-r4vdp\" (UID: \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\") " pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:17:52.982923 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.982904 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/994ef2fb-3939-4282-a7b3-e08247dbf4f0-tmp-dir\") pod \"test-model-6d272c-kserve-5589c9fc5c-r4vdp\" (UID: \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\") " pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:17:52.983059 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.982976 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/994ef2fb-3939-4282-a7b3-e08247dbf4f0-home\") pod \"test-model-6d272c-kserve-5589c9fc5c-r4vdp\" (UID: \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\") " pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:17:52.984756 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.984732 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/994ef2fb-3939-4282-a7b3-e08247dbf4f0-dshm\") pod \"test-model-6d272c-kserve-5589c9fc5c-r4vdp\" (UID: \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\") " pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:17:52.984859 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.984798 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/994ef2fb-3939-4282-a7b3-e08247dbf4f0-tls-certs\") pod \"test-model-6d272c-kserve-5589c9fc5c-r4vdp\" (UID: \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\") " pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:17:52.998773 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:52.998755 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs7hj\" (UniqueName: \"kubernetes.io/projected/994ef2fb-3939-4282-a7b3-e08247dbf4f0-kube-api-access-rs7hj\") pod \"test-model-6d272c-kserve-5589c9fc5c-r4vdp\" (UID: \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\") " pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:17:53.126990 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:53.126962 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:17:53.417716 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:53.417686 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp"] Sep 14 11:17:53.418947 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:17:53.418923 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod994ef2fb_3939_4282_a7b3_e08247dbf4f0.slice/crio-5110460f12b087c92ee35ac3c48fe4d959ca436c2d952a0afbb341a9bbb55ee4 WatchSource:0}: Error finding container 5110460f12b087c92ee35ac3c48fe4d959ca436c2d952a0afbb341a9bbb55ee4: Status 404 returned error can't find the container with id 5110460f12b087c92ee35ac3c48fe4d959ca436c2d952a0afbb341a9bbb55ee4 Sep 14 11:17:53.702546 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:53.702461 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" event={"ID":"994ef2fb-3939-4282-a7b3-e08247dbf4f0","Type":"ContainerStarted","Data":"5f0ba376d65e91d8b247f3e743a1eaa46cf72f512f724b1bc0ff4aa9140aa04c"} Sep 14 11:17:53.702546 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:53.702501 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" event={"ID":"994ef2fb-3939-4282-a7b3-e08247dbf4f0","Type":"ContainerStarted","Data":"5110460f12b087c92ee35ac3c48fe4d959ca436c2d952a0afbb341a9bbb55ee4"} Sep 14 11:17:53.738409 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:53.738358 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" podStartSLOduration=1.73834196 podStartE2EDuration="1.73834196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:17:53.735450039 +0000 UTC m=+1031.798852949" watchObservedRunningTime="2026-09-14 11:17:53.73834196 +0000 UTC m=+1031.801744935" Sep 14 11:17:55.603582 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:55.603545 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8"] Sep 14 11:17:55.604019 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:55.603817 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" podUID="3538e2e2-733f-4a08-b917-9a11a4f5a4fa" containerName="maas-api" containerID="cri-o://825a602074663cf627493a59f4fa922f3aa6dd7c1b1365c52f0d3b94ad0dc1ab" gracePeriod=30 Sep 14 11:17:55.843564 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:55.843543 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" Sep 14 11:17:55.902275 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:55.902249 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c652e0cd-8n6t2"] Sep 14 11:17:55.905823 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:55.905799 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c652e0cd-8n6t2"] Sep 14 11:17:55.910192 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:55.910176 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v55lv\" (UniqueName: \"kubernetes.io/projected/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-kube-api-access-v55lv\") pod \"3538e2e2-733f-4a08-b917-9a11a4f5a4fa\" (UID: \"3538e2e2-733f-4a08-b917-9a11a4f5a4fa\") " Sep 14 11:17:55.910251 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:55.910206 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-openshift-service-ca\") pod \"3538e2e2-733f-4a08-b917-9a11a4f5a4fa\" (UID: \"3538e2e2-733f-4a08-b917-9a11a4f5a4fa\") " Sep 14 11:17:55.910298 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:55.910287 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-maas-api-tls\") pod \"3538e2e2-733f-4a08-b917-9a11a4f5a4fa\" (UID: \"3538e2e2-733f-4a08-b917-9a11a4f5a4fa\") " Sep 14 11:17:55.910346 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:55.910332 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-metrics-certs\") pod \"3538e2e2-733f-4a08-b917-9a11a4f5a4fa\" (UID: \"3538e2e2-733f-4a08-b917-9a11a4f5a4fa\") " Sep 14 11:17:55.910633 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:55.910609 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3538e2e2-733f-4a08-b917-9a11a4f5a4fa" (UID: "3538e2e2-733f-4a08-b917-9a11a4f5a4fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:17:55.910730 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:55.910683 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-openshift-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:17:55.912234 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:55.912211 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3538e2e2-733f-4a08-b917-9a11a4f5a4fa" (UID: "3538e2e2-733f-4a08-b917-9a11a4f5a4fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:17:55.912313 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:55.912246 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3538e2e2-733f-4a08-b917-9a11a4f5a4fa" (UID: "3538e2e2-733f-4a08-b917-9a11a4f5a4fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:17:55.912313 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:55.912250 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-kube-api-access-v55lv" (OuterVolumeSpecName: "kube-api-access-v55lv") pod "3538e2e2-733f-4a08-b917-9a11a4f5a4fa" (UID: "3538e2e2-733f-4a08-b917-9a11a4f5a4fa"). InnerVolumeSpecName "kube-api-access-v55lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:17:56.011906 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.011826 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-maas-api-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:17:56.011906 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.011862 2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-metrics-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:17:56.011906 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.011878 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v55lv\" (UniqueName: \"kubernetes.io/projected/3538e2e2-733f-4a08-b917-9a11a4f5a4fa-kube-api-access-v55lv\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:17:56.045830 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.045787 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r"] Sep 14 11:17:56.046570 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.046256 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" podUID="09c975f8-8c02-4961-8567-c7408c61262b" containerName="istio-proxy" containerID="cri-o://4c85a8d5c7eb4877ec86b4b428056022fd09202c9a6d76544538c58a18e9cd99" gracePeriod=30 Sep 14 11:17:56.388238 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.388200 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acaf0e1c-e863-4b84-b301-70921eec6914" path="/var/lib/kubelet/pods/acaf0e1c-e863-4b84-b301-70921eec6914/volumes" Sep 14 11:17:56.720711 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.720626 2556 generic.go:356] "Generic (PLEG): container finished" podID="3538e2e2-733f-4a08-b917-9a11a4f5a4fa" containerID="825a602074663cf627493a59f4fa922f3aa6dd7c1b1365c52f0d3b94ad0dc1ab" exitCode=0 Sep 14 11:17:56.720711 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.720679 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" Sep 14 11:17:56.721173 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.720716 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" event={"ID":"3538e2e2-733f-4a08-b917-9a11a4f5a4fa","Type":"ContainerDied","Data":"825a602074663cf627493a59f4fa922f3aa6dd7c1b1365c52f0d3b94ad0dc1ab"} Sep 14 11:17:56.721173 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.720765 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8" event={"ID":"3538e2e2-733f-4a08-b917-9a11a4f5a4fa","Type":"ContainerDied","Data":"ace83d6c179b007de8973d72d921ca534bbb91f67bf699086537bde8b1ffaa5b"} Sep 14 11:17:56.721173 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.720789 2556 scope.go:117] "RemoveContainer" containerID="825a602074663cf627493a59f4fa922f3aa6dd7c1b1365c52f0d3b94ad0dc1ab" Sep 14 11:17:56.729432 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.729415 2556 scope.go:117] "RemoveContainer" containerID="825a602074663cf627493a59f4fa922f3aa6dd7c1b1365c52f0d3b94ad0dc1ab" Sep 14 11:17:56.729716 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:17:56.729697 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"825a602074663cf627493a59f4fa922f3aa6dd7c1b1365c52f0d3b94ad0dc1ab\": container with ID starting with 825a602074663cf627493a59f4fa922f3aa6dd7c1b1365c52f0d3b94ad0dc1ab not found: ID does not exist" containerID="825a602074663cf627493a59f4fa922f3aa6dd7c1b1365c52f0d3b94ad0dc1ab" Sep 14 11:17:56.729788 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.729726 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"825a602074663cf627493a59f4fa922f3aa6dd7c1b1365c52f0d3b94ad0dc1ab"} err="failed to get container status \"825a602074663cf627493a59f4fa922f3aa6dd7c1b1365c52f0d3b94ad0dc1ab\": rpc error: code = NotFound desc = could not find container \"825a602074663cf627493a59f4fa922f3aa6dd7c1b1365c52f0d3b94ad0dc1ab\": container with ID starting with 825a602074663cf627493a59f4fa922f3aa6dd7c1b1365c52f0d3b94ad0dc1ab not found: ID does not exist" Sep 14 11:17:56.748788 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.748764 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8"] Sep 14 11:17:56.754585 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.754562 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c652e0cd-65487bd99b-w2cl8"] Sep 14 11:17:56.790909 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.790887 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh"] Sep 14 11:17:56.791238 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.791225 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3538e2e2-733f-4a08-b917-9a11a4f5a4fa" containerName="maas-api" Sep 14 11:17:56.791278 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.791239 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3538e2e2-733f-4a08-b917-9a11a4f5a4fa" containerName="maas-api" Sep 14 11:17:56.791314 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.791307 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3538e2e2-733f-4a08-b917-9a11a4f5a4fa" containerName="maas-api" Sep 14 11:17:56.793905 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.793884 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:56.797347 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.797327 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-88bdd8bf-openshift-default-dockercfg-vbh78\"" Sep 14 11:17:56.813866 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.813847 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh"] Sep 14 11:17:56.921327 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.921294 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-workload-socket\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:56.921511 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.921337 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istio-token\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:56.921511 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.921356 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-workload-certs\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:56.921511 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.921374 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-credential-socket\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:56.921511 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.921441 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istiod-ca-cert\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:56.921511 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.921498 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5ccx\" (UniqueName: \"kubernetes.io/projected/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-kube-api-access-l5ccx\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:56.921706 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.921560 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istio-envoy\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:56.921706 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.921599 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istio-data\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:56.921706 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:56.921617 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istio-podinfo\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:57.022291 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.022215 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istio-envoy\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:57.022291 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.022255 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istio-data\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:57.022291 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.022272 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istio-podinfo\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:57.022549 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.022317 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-workload-socket\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:57.022549 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.022338 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istio-token\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:57.022549 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.022352 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-workload-certs\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:57.022549 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.022369 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-credential-socket\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:57.022549 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.022390 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istiod-ca-cert\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:57.022549 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.022421 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5ccx\" (UniqueName: \"kubernetes.io/projected/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-kube-api-access-l5ccx\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:57.022877 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.022745 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istio-data\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:57.022877 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.022761 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-workload-socket\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:57.022877 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.022800 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-workload-certs\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:57.022974 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.022895 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-credential-socket\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:57.023433 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.023407 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istiod-ca-cert\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:57.024590 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.024570 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istio-envoy\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:57.024792 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.024776 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istio-podinfo\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:57.031541 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.031517 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istio-token\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:57.032859 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.032831 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5ccx\" (UniqueName: \"kubernetes.io/projected/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-kube-api-access-l5ccx\") pod \"e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:57.103477 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.103458 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:57.397660 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:17:57.397620 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d60b178_f2ea_4d9d_b50e_4b87b58d116f.slice/crio-ea2bfce4c2f17d57513bce104cedb0c59b72a5706de537e1d9f823753583d61a WatchSource:0}: Error finding container ea2bfce4c2f17d57513bce104cedb0c59b72a5706de537e1d9f823753583d61a: Status 404 returned error can't find the container with id ea2bfce4c2f17d57513bce104cedb0c59b72a5706de537e1d9f823753583d61a Sep 14 11:17:57.397884 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.397866 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh"] Sep 14 11:17:57.400293 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.400265 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:17:57.400364 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.400324 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:17:57.400364 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.400353 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:17:57.727929 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.727841 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" event={"ID":"0d60b178-f2ea-4d9d-b50e-4b87b58d116f","Type":"ContainerStarted","Data":"36231a5f9cdd75be7cb3eff8b447212839b3ca3092442619bee9829d2be31698"} Sep 14 11:17:57.727929 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.727895 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" event={"ID":"0d60b178-f2ea-4d9d-b50e-4b87b58d116f","Type":"ContainerStarted","Data":"ea2bfce4c2f17d57513bce104cedb0c59b72a5706de537e1d9f823753583d61a"} Sep 14 11:17:57.776446 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:57.776392 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" podStartSLOduration=1.7763757089999999 podStartE2EDuration="1.776375709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:17:57.771526308 +0000 UTC m=+1035.834929217" watchObservedRunningTime="2026-09-14 11:17:57.776375709 +0000 UTC m=+1035.839778618" Sep 14 11:17:58.104238 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:58.104168 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:58.109111 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:58.109068 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:58.383023 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:58.382987 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3538e2e2-733f-4a08-b917-9a11a4f5a4fa" path="/var/lib/kubelet/pods/3538e2e2-733f-4a08-b917-9a11a4f5a4fa/volumes" Sep 14 11:17:58.733049 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:58.732971 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:17:58.733806 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:17:58.733786 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:18:01.323846 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.323762 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-27a1a2-l92h7"] Sep 14 11:18:01.330723 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.330695 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-27a1a2-l92h7"] Sep 14 11:18:01.347914 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.347894 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:18:01.465926 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.465893 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-workload-certs\") pod \"09c975f8-8c02-4961-8567-c7408c61262b\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " Sep 14 11:18:01.466114 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.465939 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c975f8-8c02-4961-8567-c7408c61262b-istiod-ca-cert\") pod \"09c975f8-8c02-4961-8567-c7408c61262b\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " Sep 14 11:18:01.466114 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.465966 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-workload-socket\") pod \"09c975f8-8c02-4961-8567-c7408c61262b\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " Sep 14 11:18:01.466114 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.466036 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c975f8-8c02-4961-8567-c7408c61262b-istio-podinfo\") pod \"09c975f8-8c02-4961-8567-c7408c61262b\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " Sep 14 11:18:01.466114 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.466109 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-istio-data\") pod \"09c975f8-8c02-4961-8567-c7408c61262b\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " Sep 14 11:18:01.466484 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.466151 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c975f8-8c02-4961-8567-c7408c61262b-istio-token\") pod \"09c975f8-8c02-4961-8567-c7408c61262b\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " Sep 14 11:18:01.466484 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.466179 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-credential-socket\") pod \"09c975f8-8c02-4961-8567-c7408c61262b\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " Sep 14 11:18:01.466484 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.466201 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "09c975f8-8c02-4961-8567-c7408c61262b" (UID: "09c975f8-8c02-4961-8567-c7408c61262b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:18:01.466484 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.466218 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "09c975f8-8c02-4961-8567-c7408c61262b" (UID: "09c975f8-8c02-4961-8567-c7408c61262b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:18:01.466484 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.466231 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j897\" (UniqueName: \"kubernetes.io/projected/09c975f8-8c02-4961-8567-c7408c61262b-kube-api-access-6j897\") pod \"09c975f8-8c02-4961-8567-c7408c61262b\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " Sep 14 11:18:01.466484 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.466269 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-istio-envoy\") pod \"09c975f8-8c02-4961-8567-c7408c61262b\" (UID: \"09c975f8-8c02-4961-8567-c7408c61262b\") " Sep 14 11:18:01.466484 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.466330 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c975f8-8c02-4961-8567-c7408c61262b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "09c975f8-8c02-4961-8567-c7408c61262b" (UID: "09c975f8-8c02-4961-8567-c7408c61262b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:18:01.466484 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.466370 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "09c975f8-8c02-4961-8567-c7408c61262b" (UID: "09c975f8-8c02-4961-8567-c7408c61262b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:18:01.466484 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.466399 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-istio-data" (OuterVolumeSpecName: "istio-data") pod "09c975f8-8c02-4961-8567-c7408c61262b" (UID: "09c975f8-8c02-4961-8567-c7408c61262b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:18:01.466922 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.466633 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:01.466922 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.466655 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:01.466922 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.466669 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:01.466922 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.466683 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c975f8-8c02-4961-8567-c7408c61262b-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:01.466922 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.466696 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:01.468353 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.468332 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09c975f8-8c02-4961-8567-c7408c61262b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "09c975f8-8c02-4961-8567-c7408c61262b" (UID: "09c975f8-8c02-4961-8567-c7408c61262b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 11:18:01.468450 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.468368 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "09c975f8-8c02-4961-8567-c7408c61262b" (UID: "09c975f8-8c02-4961-8567-c7408c61262b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:18:01.468539 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.468522 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c975f8-8c02-4961-8567-c7408c61262b-kube-api-access-6j897" (OuterVolumeSpecName: "kube-api-access-6j897") pod "09c975f8-8c02-4961-8567-c7408c61262b" (UID: "09c975f8-8c02-4961-8567-c7408c61262b"). InnerVolumeSpecName "kube-api-access-6j897". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:18:01.468750 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.468730 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c975f8-8c02-4961-8567-c7408c61262b-istio-token" (OuterVolumeSpecName: "istio-token") pod "09c975f8-8c02-4961-8567-c7408c61262b" (UID: "09c975f8-8c02-4961-8567-c7408c61262b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:18:01.567683 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.567655 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c975f8-8c02-4961-8567-c7408c61262b-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:01.567683 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.567680 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c975f8-8c02-4961-8567-c7408c61262b-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:01.567846 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.567690 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j897\" (UniqueName: \"kubernetes.io/projected/09c975f8-8c02-4961-8567-c7408c61262b-kube-api-access-6j897\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:01.567846 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.567699 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c975f8-8c02-4961-8567-c7408c61262b-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:01.749423 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.749390 2556 generic.go:356] "Generic (PLEG): container finished" podID="09c975f8-8c02-4961-8567-c7408c61262b" containerID="4c85a8d5c7eb4877ec86b4b428056022fd09202c9a6d76544538c58a18e9cd99" exitCode=0 Sep 14 11:18:01.749588 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.749434 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" Sep 14 11:18:01.749588 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.749474 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" event={"ID":"09c975f8-8c02-4961-8567-c7408c61262b","Type":"ContainerDied","Data":"4c85a8d5c7eb4877ec86b4b428056022fd09202c9a6d76544538c58a18e9cd99"} Sep 14 11:18:01.749588 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.749529 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r" event={"ID":"09c975f8-8c02-4961-8567-c7408c61262b","Type":"ContainerDied","Data":"696488f9a0090b71e2a658c98cab631a6ddec38334cab89414bdc624ae4a1713"} Sep 14 11:18:01.749588 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.749556 2556 scope.go:117] "RemoveContainer" containerID="4c85a8d5c7eb4877ec86b4b428056022fd09202c9a6d76544538c58a18e9cd99" Sep 14 11:18:01.759201 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.759182 2556 scope.go:117] "RemoveContainer" containerID="4744ac8681cf44424162fdbf1f9a93b8a78ac820b07aa0c6a34093b62de533cb" Sep 14 11:18:01.767129 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.767111 2556 scope.go:117] "RemoveContainer" containerID="4c85a8d5c7eb4877ec86b4b428056022fd09202c9a6d76544538c58a18e9cd99" Sep 14 11:18:01.767393 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:18:01.767375 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c85a8d5c7eb4877ec86b4b428056022fd09202c9a6d76544538c58a18e9cd99\": container with ID starting with 4c85a8d5c7eb4877ec86b4b428056022fd09202c9a6d76544538c58a18e9cd99 not found: ID does not exist" containerID="4c85a8d5c7eb4877ec86b4b428056022fd09202c9a6d76544538c58a18e9cd99" Sep 14 11:18:01.767440 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.767403 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c85a8d5c7eb4877ec86b4b428056022fd09202c9a6d76544538c58a18e9cd99"} err="failed to get container status \"4c85a8d5c7eb4877ec86b4b428056022fd09202c9a6d76544538c58a18e9cd99\": rpc error: code = NotFound desc = could not find container \"4c85a8d5c7eb4877ec86b4b428056022fd09202c9a6d76544538c58a18e9cd99\": container with ID starting with 4c85a8d5c7eb4877ec86b4b428056022fd09202c9a6d76544538c58a18e9cd99 not found: ID does not exist" Sep 14 11:18:01.767440 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.767419 2556 scope.go:117] "RemoveContainer" containerID="4744ac8681cf44424162fdbf1f9a93b8a78ac820b07aa0c6a34093b62de533cb" Sep 14 11:18:01.767709 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:18:01.767689 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4744ac8681cf44424162fdbf1f9a93b8a78ac820b07aa0c6a34093b62de533cb\": container with ID starting with 4744ac8681cf44424162fdbf1f9a93b8a78ac820b07aa0c6a34093b62de533cb not found: ID does not exist" containerID="4744ac8681cf44424162fdbf1f9a93b8a78ac820b07aa0c6a34093b62de533cb" Sep 14 11:18:01.767756 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.767715 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4744ac8681cf44424162fdbf1f9a93b8a78ac820b07aa0c6a34093b62de533cb"} err="failed to get container status \"4744ac8681cf44424162fdbf1f9a93b8a78ac820b07aa0c6a34093b62de533cb\": rpc error: code = NotFound desc = could not find container \"4744ac8681cf44424162fdbf1f9a93b8a78ac820b07aa0c6a34093b62de533cb\": container with ID starting with 4744ac8681cf44424162fdbf1f9a93b8a78ac820b07aa0c6a34093b62de533cb not found: ID does not exist" Sep 14 11:18:01.790169 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.790143 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r"] Sep 14 11:18:01.820031 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:01.820003 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-c652e0cd-gw-openshift-default-fd89b96b4-cc96r"] Sep 14 11:18:02.385075 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:02.385037 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09c975f8-8c02-4961-8567-c7408c61262b" path="/var/lib/kubelet/pods/09c975f8-8c02-4961-8567-c7408c61262b/volumes" Sep 14 11:18:02.385583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:02.385532 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bf74786-a1ab-44b2-8f28-c8817abb20c3" path="/var/lib/kubelet/pods/9bf74786-a1ab-44b2-8f28-c8817abb20c3/volumes" Sep 14 11:18:03.127315 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:03.127272 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:18:03.127548 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:03.127342 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:18:03.139556 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:03.139533 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:18:03.770733 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:03.770709 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:18:24.867856 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:24.867823 2556 generic.go:356] "Generic (PLEG): container finished" podID="0d60b178-f2ea-4d9d-b50e-4b87b58d116f" containerID="36231a5f9cdd75be7cb3eff8b447212839b3ca3092442619bee9829d2be31698" exitCode=0 Sep 14 11:18:24.868387 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:24.867952 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" event={"ID":"0d60b178-f2ea-4d9d-b50e-4b87b58d116f","Type":"ContainerDied","Data":"36231a5f9cdd75be7cb3eff8b447212839b3ca3092442619bee9829d2be31698"} Sep 14 11:18:24.868639 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:24.868452 2556 scope.go:117] "RemoveContainer" containerID="36231a5f9cdd75be7cb3eff8b447212839b3ca3092442619bee9829d2be31698" Sep 14 11:18:24.870116 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:24.869897 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:18:24.870116 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:24.869974 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:18:24.870116 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:24.870009 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:18:25.874942 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:25.874911 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" event={"ID":"0d60b178-f2ea-4d9d-b50e-4b87b58d116f","Type":"ContainerStarted","Data":"ba6bf4e8abda80d7806369e9ff808a7f6400adcc094daf78171c3bc3f7fd3453"} Sep 14 11:18:26.103886 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:26.103848 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:18:26.105448 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:26.105416 2556 patch_prober.go:28] interesting pod/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 14 11:18:26.105600 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:26.105484 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" podUID="0d60b178-f2ea-4d9d-b50e-4b87b58d116f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 14 11:18:27.104413 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:27.104370 2556 patch_prober.go:28] interesting pod/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 14 11:18:27.104886 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:27.104447 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" podUID="0d60b178-f2ea-4d9d-b50e-4b87b58d116f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 14 11:18:28.104511 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:28.104472 2556 patch_prober.go:28] interesting pod/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 14 11:18:28.105000 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:28.104534 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" podUID="0d60b178-f2ea-4d9d-b50e-4b87b58d116f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 14 11:18:28.733689 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:28.733658 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:18:29.104363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:29.104270 2556 patch_prober.go:28] interesting pod/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 14 11:18:29.104363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:29.104336 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" podUID="0d60b178-f2ea-4d9d-b50e-4b87b58d116f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 14 11:18:30.032834 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:30.032791 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88bdd8bf-tl2rt"] Sep 14 11:18:30.033322 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:30.033183 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09c975f8-8c02-4961-8567-c7408c61262b" containerName="istio-proxy" Sep 14 11:18:30.033322 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:30.033199 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="09c975f8-8c02-4961-8567-c7408c61262b" containerName="istio-proxy" Sep 14 11:18:30.033322 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:30.033260 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="09c975f8-8c02-4961-8567-c7408c61262b" containerName="istio-proxy" Sep 14 11:18:30.035307 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:30.035290 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88bdd8bf-tl2rt" Sep 14 11:18:30.038357 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:30.038336 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f98vt\"" Sep 14 11:18:30.038768 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:30.038755 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 11:18:30.046378 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:30.046353 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88bdd8bf-tl2rt"] Sep 14 11:18:30.104845 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:30.104441 2556 patch_prober.go:28] interesting pod/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 14 11:18:30.104845 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:30.104495 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" podUID="0d60b178-f2ea-4d9d-b50e-4b87b58d116f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 14 11:18:30.133838 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:30.133813 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02d19b30-0c0b-4045-922c-9e688c161214-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-88bdd8bf-tl2rt\" (UID: \"02d19b30-0c0b-4045-922c-9e688c161214\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88bdd8bf-tl2rt" Sep 14 11:18:30.235253 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:30.235228 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02d19b30-0c0b-4045-922c-9e688c161214-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-88bdd8bf-tl2rt\" (UID: \"02d19b30-0c0b-4045-922c-9e688c161214\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88bdd8bf-tl2rt" Sep 14 11:18:30.235815 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:30.235796 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02d19b30-0c0b-4045-922c-9e688c161214-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-88bdd8bf-tl2rt\" (UID: \"02d19b30-0c0b-4045-922c-9e688c161214\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88bdd8bf-tl2rt" Sep 14 11:18:30.345797 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:30.345735 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88bdd8bf-tl2rt" Sep 14 11:18:30.642937 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:30.642879 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88bdd8bf-tl2rt"] Sep 14 11:18:30.645330 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:18:30.645301 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02d19b30_0c0b_4045_922c_9e688c161214.slice/crio-60ef94c9ee6c46bca76cc29432e7038193e32a3f33759bce6bddf5456d7c2680 WatchSource:0}: Error finding container 60ef94c9ee6c46bca76cc29432e7038193e32a3f33759bce6bddf5456d7c2680: Status 404 returned error can't find the container with id 60ef94c9ee6c46bca76cc29432e7038193e32a3f33759bce6bddf5456d7c2680 Sep 14 11:18:30.928165 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:30.928127 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88bdd8bf-tl2rt" event={"ID":"02d19b30-0c0b-4045-922c-9e688c161214","Type":"ContainerStarted","Data":"8d2bab81a9bf61886aa13f0aab463c1739959a9e74ee0d9fe1af863aab3bc04b"} Sep 14 11:18:30.928317 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:30.928186 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88bdd8bf-tl2rt" event={"ID":"02d19b30-0c0b-4045-922c-9e688c161214","Type":"ContainerStarted","Data":"60ef94c9ee6c46bca76cc29432e7038193e32a3f33759bce6bddf5456d7c2680"} Sep 14 11:18:30.951128 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:30.951044 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88bdd8bf-tl2rt" podStartSLOduration=0.95102114 podStartE2EDuration="951.02114ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:18:30.949023133 +0000 UTC m=+1069.012426042" watchObservedRunningTime="2026-09-14 11:18:30.95102114 +0000 UTC m=+1069.014424028" Sep 14 11:18:31.107626 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:31.107597 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:18:31.108542 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:31.108519 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:18:31.934640 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:31.934612 2556 generic.go:356] "Generic (PLEG): container finished" podID="02d19b30-0c0b-4045-922c-9e688c161214" containerID="8d2bab81a9bf61886aa13f0aab463c1739959a9e74ee0d9fe1af863aab3bc04b" exitCode=0 Sep 14 11:18:31.934818 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:31.934692 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88bdd8bf-tl2rt" event={"ID":"02d19b30-0c0b-4045-922c-9e688c161214","Type":"ContainerDied","Data":"8d2bab81a9bf61886aa13f0aab463c1739959a9e74ee0d9fe1af863aab3bc04b"} Sep 14 11:18:33.070927 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:33.070905 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88bdd8bf-tl2rt" Sep 14 11:18:33.163330 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:33.163302 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02d19b30-0c0b-4045-922c-9e688c161214-maas-api-service-ca\") pod \"02d19b30-0c0b-4045-922c-9e688c161214\" (UID: \"02d19b30-0c0b-4045-922c-9e688c161214\") " Sep 14 11:18:33.163639 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:33.163617 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02d19b30-0c0b-4045-922c-9e688c161214-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "02d19b30-0c0b-4045-922c-9e688c161214" (UID: "02d19b30-0c0b-4045-922c-9e688c161214"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:18:33.265082 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:33.265002 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02d19b30-0c0b-4045-922c-9e688c161214-maas-api-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:33.945456 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:33.945423 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88bdd8bf-tl2rt" event={"ID":"02d19b30-0c0b-4045-922c-9e688c161214","Type":"ContainerDied","Data":"60ef94c9ee6c46bca76cc29432e7038193e32a3f33759bce6bddf5456d7c2680"} Sep 14 11:18:33.945624 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:33.945463 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88bdd8bf-tl2rt" Sep 14 11:18:33.945624 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:33.945469 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60ef94c9ee6c46bca76cc29432e7038193e32a3f33759bce6bddf5456d7c2680" Sep 14 11:18:41.671584 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:41.671489 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88bdd8bf-tl2rt"] Sep 14 11:18:41.673517 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:41.673494 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88bdd8bf-tl2rt"] Sep 14 11:18:42.390648 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:42.390617 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02d19b30-0c0b-4045-922c-9e688c161214" path="/var/lib/kubelet/pods/02d19b30-0c0b-4045-922c-9e688c161214/volumes" Sep 14 11:18:42.835595 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:42.835518 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh"] Sep 14 11:18:42.835989 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:42.835800 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" podUID="0d60b178-f2ea-4d9d-b50e-4b87b58d116f" containerName="istio-proxy" containerID="cri-o://ba6bf4e8abda80d7806369e9ff808a7f6400adcc094daf78171c3bc3f7fd3453" gracePeriod=30 Sep 14 11:18:43.627409 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.627374 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk"] Sep 14 11:18:43.627706 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.627693 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09c975f8-8c02-4961-8567-c7408c61262b" containerName="istio-proxy" Sep 14 11:18:43.627750 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.627707 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="09c975f8-8c02-4961-8567-c7408c61262b" containerName="istio-proxy" Sep 14 11:18:43.627750 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.627725 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02d19b30-0c0b-4045-922c-9e688c161214" containerName="revoke-keys" Sep 14 11:18:43.627750 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.627730 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="02d19b30-0c0b-4045-922c-9e688c161214" containerName="revoke-keys" Sep 14 11:18:43.627857 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.627786 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="02d19b30-0c0b-4045-922c-9e688c161214" containerName="revoke-keys" Sep 14 11:18:43.627857 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.627794 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="09c975f8-8c02-4961-8567-c7408c61262b" containerName="istio-proxy" Sep 14 11:18:43.630752 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.630736 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.634242 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.634224 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-6fcec1be-openshift-default-dockercfg-544nf\"" Sep 14 11:18:43.651326 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.651306 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk"] Sep 14 11:18:43.761238 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.761205 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7kkg\" (UniqueName: \"kubernetes.io/projected/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-kube-api-access-k7kkg\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.761415 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.761321 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istio-envoy\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.761415 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.761386 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istio-token\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.761501 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.761461 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-credential-socket\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.761549 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.761496 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-workload-socket\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.761587 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.761562 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istio-data\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.761633 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.761592 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-workload-certs\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.761633 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.761621 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istiod-ca-cert\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.761707 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.761652 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istio-podinfo\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.862202 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.862173 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istio-data\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.862612 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.862208 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-workload-certs\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.862612 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.862229 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istiod-ca-cert\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.862612 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.862259 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istio-podinfo\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.862612 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.862294 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7kkg\" (UniqueName: \"kubernetes.io/projected/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-kube-api-access-k7kkg\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.862612 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.862360 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istio-envoy\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.862612 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.862426 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istio-token\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.862612 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.862480 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-credential-socket\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.862612 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.862506 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-workload-socket\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.862612 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.862589 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-workload-certs\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.863127 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.862645 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istio-data\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.863127 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.862779 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-workload-socket\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.863127 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.862886 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-credential-socket\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.863127 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.863022 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istiod-ca-cert\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.865010 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.864986 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istio-envoy\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.865215 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.865200 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istio-podinfo\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.874280 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.874257 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7kkg\" (UniqueName: \"kubernetes.io/projected/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-kube-api-access-k7kkg\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.874556 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.874539 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istio-token\") pod \"e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:43.940676 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:43.940617 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:44.217220 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:44.217196 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk"] Sep 14 11:18:44.218795 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:18:44.218771 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd634c091_026a_4f44_8ba6_5cd47b8e2d8a.slice/crio-5e6bcbdebf66dc125d5dbaf252c1b400be44ab24c80846ee24c44c0e873900f1 WatchSource:0}: Error finding container 5e6bcbdebf66dc125d5dbaf252c1b400be44ab24c80846ee24c44c0e873900f1: Status 404 returned error can't find the container with id 5e6bcbdebf66dc125d5dbaf252c1b400be44ab24c80846ee24c44c0e873900f1 Sep 14 11:18:44.220997 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:44.220963 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:18:44.221071 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:44.221027 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:18:44.221071 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:44.221055 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:18:45.003231 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:45.003196 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" event={"ID":"d634c091-026a-4f44-8ba6-5cd47b8e2d8a","Type":"ContainerStarted","Data":"928601cbb9768c18bde3bbc7f10c33d4d74a5eb213d640394f1e1ea3edc5c220"} Sep 14 11:18:45.003570 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:45.003245 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" event={"ID":"d634c091-026a-4f44-8ba6-5cd47b8e2d8a","Type":"ContainerStarted","Data":"5e6bcbdebf66dc125d5dbaf252c1b400be44ab24c80846ee24c44c0e873900f1"} Sep 14 11:18:45.029218 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:45.029174 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" podStartSLOduration=2.029159433 podStartE2EDuration="2.029159433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:18:45.027538238 +0000 UTC m=+1083.090941148" watchObservedRunningTime="2026-09-14 11:18:45.029159433 +0000 UTC m=+1083.092562343" Sep 14 11:18:45.825671 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:45.825640 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd"] Sep 14 11:18:45.825892 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:45.825871 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" podUID="3af40be6-bb3e-472d-bec5-eaf171cc900d" containerName="istio-proxy" containerID="cri-o://492c8b18b062c7b56820bedf91066cdc959d9799c62045c95a4dc11a639b97cf" gracePeriod=30 Sep 14 11:18:45.943207 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:45.943167 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:45.953268 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:45.953237 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:46.008325 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:46.008292 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:46.009204 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:46.009186 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:18:46.110181 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:46.110153 2556 patch_prober.go:28] interesting pod/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 14 11:18:46.110333 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:46.110203 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" podUID="0d60b178-f2ea-4d9d-b50e-4b87b58d116f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 14 11:18:48.019769 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.019742 2556 generic.go:356] "Generic (PLEG): container finished" podID="0d60b178-f2ea-4d9d-b50e-4b87b58d116f" containerID="ba6bf4e8abda80d7806369e9ff808a7f6400adcc094daf78171c3bc3f7fd3453" exitCode=0 Sep 14 11:18:48.020179 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.019820 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" event={"ID":"0d60b178-f2ea-4d9d-b50e-4b87b58d116f","Type":"ContainerDied","Data":"ba6bf4e8abda80d7806369e9ff808a7f6400adcc094daf78171c3bc3f7fd3453"} Sep 14 11:18:48.020179 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.019901 2556 scope.go:117] "RemoveContainer" containerID="36231a5f9cdd75be7cb3eff8b447212839b3ca3092442619bee9829d2be31698" Sep 14 11:18:48.077534 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.077514 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:18:48.205771 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.205737 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istio-podinfo\") pod \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " Sep 14 11:18:48.205771 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.205778 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-workload-certs\") pod \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " Sep 14 11:18:48.205950 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.205805 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istio-token\") pod \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " Sep 14 11:18:48.205950 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.205827 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-credential-socket\") pod \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " Sep 14 11:18:48.205950 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.205883 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istio-envoy\") pod \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " Sep 14 11:18:48.205950 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.205933 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istiod-ca-cert\") pod \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " Sep 14 11:18:48.206178 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.205996 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istio-data\") pod \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " Sep 14 11:18:48.206178 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.206015 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0d60b178-f2ea-4d9d-b50e-4b87b58d116f" (UID: "0d60b178-f2ea-4d9d-b50e-4b87b58d116f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:18:48.206178 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.206023 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-workload-socket\") pod \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " Sep 14 11:18:48.206178 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.206104 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5ccx\" (UniqueName: \"kubernetes.io/projected/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-kube-api-access-l5ccx\") pod \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\" (UID: \"0d60b178-f2ea-4d9d-b50e-4b87b58d116f\") " Sep 14 11:18:48.206389 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.206247 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istio-data" (OuterVolumeSpecName: "istio-data") pod "0d60b178-f2ea-4d9d-b50e-4b87b58d116f" (UID: "0d60b178-f2ea-4d9d-b50e-4b87b58d116f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:18:48.206389 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.206296 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0d60b178-f2ea-4d9d-b50e-4b87b58d116f" (UID: "0d60b178-f2ea-4d9d-b50e-4b87b58d116f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:18:48.206389 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.206371 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0d60b178-f2ea-4d9d-b50e-4b87b58d116f" (UID: "0d60b178-f2ea-4d9d-b50e-4b87b58d116f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:18:48.206542 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.206456 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0d60b178-f2ea-4d9d-b50e-4b87b58d116f" (UID: "0d60b178-f2ea-4d9d-b50e-4b87b58d116f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:18:48.206601 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.206587 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:48.206653 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.206608 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:48.206653 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.206620 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:48.206653 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.206633 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:48.206653 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.206646 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:48.207934 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.207911 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0d60b178-f2ea-4d9d-b50e-4b87b58d116f" (UID: "0d60b178-f2ea-4d9d-b50e-4b87b58d116f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 11:18:48.208112 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.208068 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istio-token" (OuterVolumeSpecName: "istio-token") pod "0d60b178-f2ea-4d9d-b50e-4b87b58d116f" (UID: "0d60b178-f2ea-4d9d-b50e-4b87b58d116f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:18:48.208172 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.208146 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-kube-api-access-l5ccx" (OuterVolumeSpecName: "kube-api-access-l5ccx") pod "0d60b178-f2ea-4d9d-b50e-4b87b58d116f" (UID: "0d60b178-f2ea-4d9d-b50e-4b87b58d116f"). InnerVolumeSpecName "kube-api-access-l5ccx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:18:48.208225 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.208209 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0d60b178-f2ea-4d9d-b50e-4b87b58d116f" (UID: "0d60b178-f2ea-4d9d-b50e-4b87b58d116f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:18:48.307914 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.307887 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5ccx\" (UniqueName: \"kubernetes.io/projected/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-kube-api-access-l5ccx\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:48.307914 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.307911 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:48.308027 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.307920 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:48.308027 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:48.307927 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d60b178-f2ea-4d9d-b50e-4b87b58d116f-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:49.025843 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:49.025813 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" Sep 14 11:18:49.026273 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:49.025851 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh" event={"ID":"0d60b178-f2ea-4d9d-b50e-4b87b58d116f","Type":"ContainerDied","Data":"ea2bfce4c2f17d57513bce104cedb0c59b72a5706de537e1d9f823753583d61a"} Sep 14 11:18:49.026273 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:49.025918 2556 scope.go:117] "RemoveContainer" containerID="ba6bf4e8abda80d7806369e9ff808a7f6400adcc094daf78171c3bc3f7fd3453" Sep 14 11:18:49.045905 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:49.045882 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh"] Sep 14 11:18:49.049964 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:49.049935 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-88bdd8bf-openshift-default-59665ccf44-gqkxh"] Sep 14 11:18:50.387750 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:50.387713 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d60b178-f2ea-4d9d-b50e-4b87b58d116f" path="/var/lib/kubelet/pods/0d60b178-f2ea-4d9d-b50e-4b87b58d116f/volumes" Sep 14 11:18:51.039549 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.039518 2556 generic.go:356] "Generic (PLEG): container finished" podID="3af40be6-bb3e-472d-bec5-eaf171cc900d" containerID="492c8b18b062c7b56820bedf91066cdc959d9799c62045c95a4dc11a639b97cf" exitCode=0 Sep 14 11:18:51.039707 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.039595 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" event={"ID":"3af40be6-bb3e-472d-bec5-eaf171cc900d","Type":"ContainerDied","Data":"492c8b18b062c7b56820bedf91066cdc959d9799c62045c95a4dc11a639b97cf"} Sep 14 11:18:51.039707 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.039667 2556 scope.go:117] "RemoveContainer" containerID="771f7c18cedea400923dde3d268a2a384bd887588ff68bbd82f6c4acec9b591d" Sep 14 11:18:51.072922 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.072897 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:18:51.234170 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.234141 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47jjl\" (UniqueName: \"kubernetes.io/projected/3af40be6-bb3e-472d-bec5-eaf171cc900d-kube-api-access-47jjl\") pod \"3af40be6-bb3e-472d-bec5-eaf171cc900d\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " Sep 14 11:18:51.234323 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.234187 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-credential-socket\") pod \"3af40be6-bb3e-472d-bec5-eaf171cc900d\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " Sep 14 11:18:51.234323 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.234237 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-istio-envoy\") pod \"3af40be6-bb3e-472d-bec5-eaf171cc900d\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " Sep 14 11:18:51.234323 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.234277 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af40be6-bb3e-472d-bec5-eaf171cc900d-istio-podinfo\") pod \"3af40be6-bb3e-472d-bec5-eaf171cc900d\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " Sep 14 11:18:51.234323 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.234314 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-workload-certs\") pod \"3af40be6-bb3e-472d-bec5-eaf171cc900d\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " Sep 14 11:18:51.234502 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.234353 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-istio-data\") pod \"3af40be6-bb3e-472d-bec5-eaf171cc900d\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " Sep 14 11:18:51.234502 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.234381 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af40be6-bb3e-472d-bec5-eaf171cc900d-istio-token\") pod \"3af40be6-bb3e-472d-bec5-eaf171cc900d\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " Sep 14 11:18:51.234502 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.234407 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-workload-socket\") pod \"3af40be6-bb3e-472d-bec5-eaf171cc900d\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " Sep 14 11:18:51.234502 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.234468 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af40be6-bb3e-472d-bec5-eaf171cc900d-istiod-ca-cert\") pod \"3af40be6-bb3e-472d-bec5-eaf171cc900d\" (UID: \"3af40be6-bb3e-472d-bec5-eaf171cc900d\") " Sep 14 11:18:51.234686 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.234530 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3af40be6-bb3e-472d-bec5-eaf171cc900d" (UID: "3af40be6-bb3e-472d-bec5-eaf171cc900d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:18:51.234686 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.234658 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3af40be6-bb3e-472d-bec5-eaf171cc900d" (UID: "3af40be6-bb3e-472d-bec5-eaf171cc900d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:18:51.234686 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.234670 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-istio-data" (OuterVolumeSpecName: "istio-data") pod "3af40be6-bb3e-472d-bec5-eaf171cc900d" (UID: "3af40be6-bb3e-472d-bec5-eaf171cc900d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:18:51.234833 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.234758 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3af40be6-bb3e-472d-bec5-eaf171cc900d" (UID: "3af40be6-bb3e-472d-bec5-eaf171cc900d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:18:51.234881 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.234856 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:51.234936 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.234876 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:51.234936 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.234891 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:51.234936 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.234905 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:51.234936 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.234924 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af40be6-bb3e-472d-bec5-eaf171cc900d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3af40be6-bb3e-472d-bec5-eaf171cc900d" (UID: "3af40be6-bb3e-472d-bec5-eaf171cc900d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:18:51.236450 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.236422 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af40be6-bb3e-472d-bec5-eaf171cc900d-kube-api-access-47jjl" (OuterVolumeSpecName: "kube-api-access-47jjl") pod "3af40be6-bb3e-472d-bec5-eaf171cc900d" (UID: "3af40be6-bb3e-472d-bec5-eaf171cc900d"). InnerVolumeSpecName "kube-api-access-47jjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:18:51.236557 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.236444 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af40be6-bb3e-472d-bec5-eaf171cc900d-istio-token" (OuterVolumeSpecName: "istio-token") pod "3af40be6-bb3e-472d-bec5-eaf171cc900d" (UID: "3af40be6-bb3e-472d-bec5-eaf171cc900d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:18:51.236557 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.236530 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3af40be6-bb3e-472d-bec5-eaf171cc900d" (UID: "3af40be6-bb3e-472d-bec5-eaf171cc900d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:18:51.236726 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.236707 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3af40be6-bb3e-472d-bec5-eaf171cc900d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3af40be6-bb3e-472d-bec5-eaf171cc900d" (UID: "3af40be6-bb3e-472d-bec5-eaf171cc900d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 11:18:51.336360 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.336339 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af40be6-bb3e-472d-bec5-eaf171cc900d-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:51.336360 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.336359 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af40be6-bb3e-472d-bec5-eaf171cc900d-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:51.336601 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.336367 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af40be6-bb3e-472d-bec5-eaf171cc900d-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:51.336601 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.336376 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47jjl\" (UniqueName: \"kubernetes.io/projected/3af40be6-bb3e-472d-bec5-eaf171cc900d-kube-api-access-47jjl\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:51.336601 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.336385 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af40be6-bb3e-472d-bec5-eaf171cc900d-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:18:51.402606 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.402580 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq"] Sep 14 11:18:51.403071 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.403056 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d60b178-f2ea-4d9d-b50e-4b87b58d116f" containerName="istio-proxy" Sep 14 11:18:51.403138 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.403073 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d60b178-f2ea-4d9d-b50e-4b87b58d116f" containerName="istio-proxy" Sep 14 11:18:51.403138 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.403105 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3af40be6-bb3e-472d-bec5-eaf171cc900d" containerName="istio-proxy" Sep 14 11:18:51.403138 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.403114 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3af40be6-bb3e-472d-bec5-eaf171cc900d" containerName="istio-proxy" Sep 14 11:18:51.403138 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.403124 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3af40be6-bb3e-472d-bec5-eaf171cc900d" containerName="istio-proxy" Sep 14 11:18:51.403138 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.403132 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3af40be6-bb3e-472d-bec5-eaf171cc900d" containerName="istio-proxy" Sep 14 11:18:51.403285 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.403145 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d60b178-f2ea-4d9d-b50e-4b87b58d116f" containerName="istio-proxy" Sep 14 11:18:51.403285 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.403153 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d60b178-f2ea-4d9d-b50e-4b87b58d116f" containerName="istio-proxy" Sep 14 11:18:51.403285 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.403237 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3af40be6-bb3e-472d-bec5-eaf171cc900d" containerName="istio-proxy" Sep 14 11:18:51.403285 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.403247 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d60b178-f2ea-4d9d-b50e-4b87b58d116f" containerName="istio-proxy" Sep 14 11:18:51.403285 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.403258 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d60b178-f2ea-4d9d-b50e-4b87b58d116f" containerName="istio-proxy" Sep 14 11:18:51.403285 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.403265 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3af40be6-bb3e-472d-bec5-eaf171cc900d" containerName="istio-proxy" Sep 14 11:18:51.405623 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.405605 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq" Sep 14 11:18:51.408476 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.408452 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-6fcec1be-serving-cert\"" Sep 14 11:18:51.408585 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.408459 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 14 11:18:51.409642 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.409623 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 11:18:51.409986 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.409972 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w7q69\"" Sep 14 11:18:51.410344 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.410327 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 14 11:18:51.419249 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.419227 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq"] Sep 14 11:18:51.538711 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.538630 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4246d01-d173-47fb-881a-dc2594ad6a9d-maas-api-tls\") pod \"maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq\" (UID: \"d4246d01-d173-47fb-881a-dc2594ad6a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq" Sep 14 11:18:51.538711 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.538666 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr4sk\" (UniqueName: \"kubernetes.io/projected/d4246d01-d173-47fb-881a-dc2594ad6a9d-kube-api-access-nr4sk\") pod \"maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq\" (UID: \"d4246d01-d173-47fb-881a-dc2594ad6a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq" Sep 14 11:18:51.538895 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.538751 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4246d01-d173-47fb-881a-dc2594ad6a9d-metrics-certs\") pod \"maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq\" (UID: \"d4246d01-d173-47fb-881a-dc2594ad6a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq" Sep 14 11:18:51.538895 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.538851 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4246d01-d173-47fb-881a-dc2594ad6a9d-openshift-service-ca\") pod \"maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq\" (UID: \"d4246d01-d173-47fb-881a-dc2594ad6a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq" Sep 14 11:18:51.640269 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.640231 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4246d01-d173-47fb-881a-dc2594ad6a9d-openshift-service-ca\") pod \"maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq\" (UID: \"d4246d01-d173-47fb-881a-dc2594ad6a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq" Sep 14 11:18:51.640417 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.640285 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4246d01-d173-47fb-881a-dc2594ad6a9d-maas-api-tls\") pod \"maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq\" (UID: \"d4246d01-d173-47fb-881a-dc2594ad6a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq" Sep 14 11:18:51.640417 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.640306 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr4sk\" (UniqueName: \"kubernetes.io/projected/d4246d01-d173-47fb-881a-dc2594ad6a9d-kube-api-access-nr4sk\") pod \"maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq\" (UID: \"d4246d01-d173-47fb-881a-dc2594ad6a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq" Sep 14 11:18:51.640417 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.640348 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4246d01-d173-47fb-881a-dc2594ad6a9d-metrics-certs\") pod \"maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq\" (UID: \"d4246d01-d173-47fb-881a-dc2594ad6a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq" Sep 14 11:18:51.640914 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.640895 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4246d01-d173-47fb-881a-dc2594ad6a9d-openshift-service-ca\") pod \"maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq\" (UID: \"d4246d01-d173-47fb-881a-dc2594ad6a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq" Sep 14 11:18:51.642726 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.642697 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4246d01-d173-47fb-881a-dc2594ad6a9d-metrics-certs\") pod \"maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq\" (UID: \"d4246d01-d173-47fb-881a-dc2594ad6a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq" Sep 14 11:18:51.642831 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.642745 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4246d01-d173-47fb-881a-dc2594ad6a9d-maas-api-tls\") pod \"maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq\" (UID: \"d4246d01-d173-47fb-881a-dc2594ad6a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq" Sep 14 11:18:51.653519 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.653496 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr4sk\" (UniqueName: \"kubernetes.io/projected/d4246d01-d173-47fb-881a-dc2594ad6a9d-kube-api-access-nr4sk\") pod \"maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq\" (UID: \"d4246d01-d173-47fb-881a-dc2594ad6a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq" Sep 14 11:18:51.716070 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.716043 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq" Sep 14 11:18:51.991273 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:51.991242 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq"] Sep 14 11:18:51.994881 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:18:51.994843 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4246d01_d173_47fb_881a_dc2594ad6a9d.slice/crio-d0b834152336d04fb00caad6194b58a8dcc825692ff67656535649ba924963e6 WatchSource:0}: Error finding container d0b834152336d04fb00caad6194b58a8dcc825692ff67656535649ba924963e6: Status 404 returned error can't find the container with id d0b834152336d04fb00caad6194b58a8dcc825692ff67656535649ba924963e6 Sep 14 11:18:52.045865 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:52.045844 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" Sep 14 11:18:52.045984 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:52.045884 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd" event={"ID":"3af40be6-bb3e-472d-bec5-eaf171cc900d","Type":"ContainerDied","Data":"bd216b59173fb983118c79197ae94fea53182d1047333587c2bf8be5a81ccee7"} Sep 14 11:18:52.045984 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:52.045954 2556 scope.go:117] "RemoveContainer" containerID="492c8b18b062c7b56820bedf91066cdc959d9799c62045c95a4dc11a639b97cf" Sep 14 11:18:52.047366 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:52.047341 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq" event={"ID":"d4246d01-d173-47fb-881a-dc2594ad6a9d","Type":"ContainerStarted","Data":"d0b834152336d04fb00caad6194b58a8dcc825692ff67656535649ba924963e6"} Sep 14 11:18:52.081191 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:52.081163 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd"] Sep 14 11:18:52.083529 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:52.083507 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-27a1a2-openshift-default-76dbcbcd9d-cs8fd"] Sep 14 11:18:52.382559 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:52.382523 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3af40be6-bb3e-472d-bec5-eaf171cc900d" path="/var/lib/kubelet/pods/3af40be6-bb3e-472d-bec5-eaf171cc900d/volumes" Sep 14 11:18:53.055775 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:53.055742 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq" event={"ID":"d4246d01-d173-47fb-881a-dc2594ad6a9d","Type":"ContainerStarted","Data":"d3cff781a340c00491b095f9e7c97dbee23edd19b16b469d9b67671c62f69e39"} Sep 14 11:18:53.056239 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:53.055864 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq" Sep 14 11:18:59.066045 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:59.066014 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq" Sep 14 11:18:59.100790 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:18:59.100720 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq" podStartSLOduration=7.214627618 podStartE2EDuration="8.100698881s" totalImagesPullingTime="886.071263ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:18:53.080042363 +0000 UTC m=+1091.143445296" watchObservedRunningTime="2026-09-14 11:18:59.100698881 +0000 UTC m=+1097.164101793" Sep 14 11:19:00.209412 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:00.209372 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn"] Sep 14 11:19:00.212628 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:00.212606 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" Sep 14 11:19:00.216637 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:00.216433 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-41158e-serving-cert\"" Sep 14 11:19:00.227925 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:00.227899 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn"] Sep 14 11:19:00.322866 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:00.322817 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/743ca772-cec4-4dc3-9537-e0d2d9aa7987-openshift-service-ca\") pod \"maas-api-e2e-explicit-41158e-558f79fd7-vnzkn\" (UID: \"743ca772-cec4-4dc3-9537-e0d2d9aa7987\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" Sep 14 11:19:00.323073 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:00.322891 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/743ca772-cec4-4dc3-9537-e0d2d9aa7987-maas-api-tls\") pod \"maas-api-e2e-explicit-41158e-558f79fd7-vnzkn\" (UID: \"743ca772-cec4-4dc3-9537-e0d2d9aa7987\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" Sep 14 11:19:00.323073 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:00.322919 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpzp7\" (UniqueName: \"kubernetes.io/projected/743ca772-cec4-4dc3-9537-e0d2d9aa7987-kube-api-access-qpzp7\") pod \"maas-api-e2e-explicit-41158e-558f79fd7-vnzkn\" (UID: \"743ca772-cec4-4dc3-9537-e0d2d9aa7987\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" Sep 14 11:19:00.323073 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:00.322986 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/743ca772-cec4-4dc3-9537-e0d2d9aa7987-metrics-certs\") pod \"maas-api-e2e-explicit-41158e-558f79fd7-vnzkn\" (UID: \"743ca772-cec4-4dc3-9537-e0d2d9aa7987\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" Sep 14 11:19:00.423605 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:00.423572 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/743ca772-cec4-4dc3-9537-e0d2d9aa7987-metrics-certs\") pod \"maas-api-e2e-explicit-41158e-558f79fd7-vnzkn\" (UID: \"743ca772-cec4-4dc3-9537-e0d2d9aa7987\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" Sep 14 11:19:00.423810 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:00.423665 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/743ca772-cec4-4dc3-9537-e0d2d9aa7987-openshift-service-ca\") pod \"maas-api-e2e-explicit-41158e-558f79fd7-vnzkn\" (UID: \"743ca772-cec4-4dc3-9537-e0d2d9aa7987\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" Sep 14 11:19:00.423810 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:00.423685 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/743ca772-cec4-4dc3-9537-e0d2d9aa7987-maas-api-tls\") pod \"maas-api-e2e-explicit-41158e-558f79fd7-vnzkn\" (UID: \"743ca772-cec4-4dc3-9537-e0d2d9aa7987\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" Sep 14 11:19:00.423810 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:00.423702 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpzp7\" (UniqueName: \"kubernetes.io/projected/743ca772-cec4-4dc3-9537-e0d2d9aa7987-kube-api-access-qpzp7\") pod \"maas-api-e2e-explicit-41158e-558f79fd7-vnzkn\" (UID: \"743ca772-cec4-4dc3-9537-e0d2d9aa7987\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" Sep 14 11:19:00.423810 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:19:00.423790 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-serving-cert: secret "maas-api-e2e-explicit-41158e-serving-cert" not found Sep 14 11:19:00.424068 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:19:00.423859 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/743ca772-cec4-4dc3-9537-e0d2d9aa7987-maas-api-tls podName:743ca772-cec4-4dc3-9537-e0d2d9aa7987 nodeName:}" failed. No retries permitted until 2026-09-14 11:19:00.923843429 +0000 UTC m=+1098.987246317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/743ca772-cec4-4dc3-9537-e0d2d9aa7987-maas-api-tls") pod "maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" (UID: "743ca772-cec4-4dc3-9537-e0d2d9aa7987") : secret "maas-api-e2e-explicit-41158e-serving-cert" not found Sep 14 11:19:00.424422 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:00.424392 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/743ca772-cec4-4dc3-9537-e0d2d9aa7987-openshift-service-ca\") pod \"maas-api-e2e-explicit-41158e-558f79fd7-vnzkn\" (UID: \"743ca772-cec4-4dc3-9537-e0d2d9aa7987\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" Sep 14 11:19:00.426259 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:00.426243 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/743ca772-cec4-4dc3-9537-e0d2d9aa7987-metrics-certs\") pod \"maas-api-e2e-explicit-41158e-558f79fd7-vnzkn\" (UID: \"743ca772-cec4-4dc3-9537-e0d2d9aa7987\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" Sep 14 11:19:00.432622 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:00.432597 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpzp7\" (UniqueName: \"kubernetes.io/projected/743ca772-cec4-4dc3-9537-e0d2d9aa7987-kube-api-access-qpzp7\") pod \"maas-api-e2e-explicit-41158e-558f79fd7-vnzkn\" (UID: \"743ca772-cec4-4dc3-9537-e0d2d9aa7987\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" Sep 14 11:19:00.927594 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:00.927561 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/743ca772-cec4-4dc3-9537-e0d2d9aa7987-maas-api-tls\") pod \"maas-api-e2e-explicit-41158e-558f79fd7-vnzkn\" (UID: \"743ca772-cec4-4dc3-9537-e0d2d9aa7987\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" Sep 14 11:19:00.929940 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:00.929914 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/743ca772-cec4-4dc3-9537-e0d2d9aa7987-maas-api-tls\") pod \"maas-api-e2e-explicit-41158e-558f79fd7-vnzkn\" (UID: \"743ca772-cec4-4dc3-9537-e0d2d9aa7987\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" Sep 14 11:19:01.125122 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:01.125074 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" Sep 14 11:19:01.406113 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:01.405645 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn"] Sep 14 11:19:01.414422 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:19:01.414384 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743ca772_cec4_4dc3_9537_e0d2d9aa7987.slice/crio-f076a3e7d357688dca253d3c5a98d2ba3d48b9027891eaca9451a5e16d52da3a WatchSource:0}: Error finding container f076a3e7d357688dca253d3c5a98d2ba3d48b9027891eaca9451a5e16d52da3a: Status 404 returned error can't find the container with id f076a3e7d357688dca253d3c5a98d2ba3d48b9027891eaca9451a5e16d52da3a Sep 14 11:19:02.110693 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:02.110652 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" event={"ID":"743ca772-cec4-4dc3-9537-e0d2d9aa7987","Type":"ContainerStarted","Data":"bf81e1e8b90be7806edda4cd42ac15edae58292eb5e5e9cf9d32927a1f311f51"} Sep 14 11:19:02.110860 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:02.110726 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" event={"ID":"743ca772-cec4-4dc3-9537-e0d2d9aa7987","Type":"ContainerStarted","Data":"f076a3e7d357688dca253d3c5a98d2ba3d48b9027891eaca9451a5e16d52da3a"} Sep 14 11:19:02.111598 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:02.111570 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" Sep 14 11:19:02.138804 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:02.138754 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" podStartSLOduration=1.786030819 podStartE2EDuration="2.138738288s" totalImagesPullingTime="352.707469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:19:02.137362606 +0000 UTC m=+1100.200765516" watchObservedRunningTime="2026-09-14 11:19:02.138738288 +0000 UTC m=+1100.202141226" Sep 14 11:19:07.139509 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:07.139473 2556 generic.go:356] "Generic (PLEG): container finished" podID="d634c091-026a-4f44-8ba6-5cd47b8e2d8a" containerID="928601cbb9768c18bde3bbc7f10c33d4d74a5eb213d640394f1e1ea3edc5c220" exitCode=0 Sep 14 11:19:07.139922 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:07.139544 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" event={"ID":"d634c091-026a-4f44-8ba6-5cd47b8e2d8a","Type":"ContainerDied","Data":"928601cbb9768c18bde3bbc7f10c33d4d74a5eb213d640394f1e1ea3edc5c220"} Sep 14 11:19:07.139922 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:07.139909 2556 scope.go:117] "RemoveContainer" containerID="928601cbb9768c18bde3bbc7f10c33d4d74a5eb213d640394f1e1ea3edc5c220" Sep 14 11:19:07.141023 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:07.140996 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:19:07.141082 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:07.141060 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:19:07.141189 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:07.141114 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:19:08.149925 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:08.149880 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" event={"ID":"d634c091-026a-4f44-8ba6-5cd47b8e2d8a","Type":"ContainerStarted","Data":"7a210b98c9ead016c070e7400152e50bf414649fa02451f51008fd1aff493332"} Sep 14 11:19:08.941785 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:08.941747 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:19:08.942122 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:08.941905 2556 patch_prober.go:28] interesting pod/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 14 11:19:08.942269 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:08.942139 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" podUID="d634c091-026a-4f44-8ba6-5cd47b8e2d8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 14 11:19:09.127294 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:09.127265 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" Sep 14 11:19:09.941928 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:09.941888 2556 patch_prober.go:28] interesting pod/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 14 11:19:09.942428 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:09.941947 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" podUID="d634c091-026a-4f44-8ba6-5cd47b8e2d8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 14 11:19:10.941762 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:10.941729 2556 patch_prober.go:28] interesting pod/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 14 11:19:10.941928 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:10.941780 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" podUID="d634c091-026a-4f44-8ba6-5cd47b8e2d8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 14 11:19:11.911318 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:11.911283 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq"] Sep 14 11:19:11.912077 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:11.912044 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq" podUID="d4246d01-d173-47fb-881a-dc2594ad6a9d" containerName="maas-api" containerID="cri-o://d3cff781a340c00491b095f9e7c97dbee23edd19b16b469d9b67671c62f69e39" gracePeriod=30 Sep 14 11:19:11.941354 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:11.941323 2556 patch_prober.go:28] interesting pod/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 14 11:19:11.941458 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:11.941372 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" podUID="d634c091-026a-4f44-8ba6-5cd47b8e2d8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 14 11:19:12.173860 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:12.173777 2556 generic.go:356] "Generic (PLEG): container finished" podID="d4246d01-d173-47fb-881a-dc2594ad6a9d" containerID="d3cff781a340c00491b095f9e7c97dbee23edd19b16b469d9b67671c62f69e39" exitCode=0 Sep 14 11:19:12.174132 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:12.173837 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq" event={"ID":"d4246d01-d173-47fb-881a-dc2594ad6a9d","Type":"ContainerDied","Data":"d3cff781a340c00491b095f9e7c97dbee23edd19b16b469d9b67671c62f69e39"} Sep 14 11:19:12.214918 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:12.214771 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq" Sep 14 11:19:12.244211 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:12.244186 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nr4sk\" (UniqueName: \"kubernetes.io/projected/d4246d01-d173-47fb-881a-dc2594ad6a9d-kube-api-access-nr4sk\") pod \"d4246d01-d173-47fb-881a-dc2594ad6a9d\" (UID: \"d4246d01-d173-47fb-881a-dc2594ad6a9d\") " Sep 14 11:19:12.244342 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:12.244261 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4246d01-d173-47fb-881a-dc2594ad6a9d-maas-api-tls\") pod \"d4246d01-d173-47fb-881a-dc2594ad6a9d\" (UID: \"d4246d01-d173-47fb-881a-dc2594ad6a9d\") " Sep 14 11:19:12.244342 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:12.244302 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4246d01-d173-47fb-881a-dc2594ad6a9d-openshift-service-ca\") pod \"d4246d01-d173-47fb-881a-dc2594ad6a9d\" (UID: \"d4246d01-d173-47fb-881a-dc2594ad6a9d\") " Sep 14 11:19:12.244342 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:12.244335 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4246d01-d173-47fb-881a-dc2594ad6a9d-metrics-certs\") pod \"d4246d01-d173-47fb-881a-dc2594ad6a9d\" (UID: \"d4246d01-d173-47fb-881a-dc2594ad6a9d\") " Sep 14 11:19:12.244682 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:12.244655 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4246d01-d173-47fb-881a-dc2594ad6a9d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d4246d01-d173-47fb-881a-dc2594ad6a9d" (UID: "d4246d01-d173-47fb-881a-dc2594ad6a9d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:19:12.246468 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:12.246392 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4246d01-d173-47fb-881a-dc2594ad6a9d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d4246d01-d173-47fb-881a-dc2594ad6a9d" (UID: "d4246d01-d173-47fb-881a-dc2594ad6a9d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:19:12.246468 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:12.246427 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4246d01-d173-47fb-881a-dc2594ad6a9d-kube-api-access-nr4sk" (OuterVolumeSpecName: "kube-api-access-nr4sk") pod "d4246d01-d173-47fb-881a-dc2594ad6a9d" (UID: "d4246d01-d173-47fb-881a-dc2594ad6a9d"). InnerVolumeSpecName "kube-api-access-nr4sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:19:12.246468 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:12.246451 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4246d01-d173-47fb-881a-dc2594ad6a9d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d4246d01-d173-47fb-881a-dc2594ad6a9d" (UID: "d4246d01-d173-47fb-881a-dc2594ad6a9d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:19:12.345605 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:12.345570 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr4sk\" (UniqueName: \"kubernetes.io/projected/d4246d01-d173-47fb-881a-dc2594ad6a9d-kube-api-access-nr4sk\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:19:12.345605 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:12.345602 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4246d01-d173-47fb-881a-dc2594ad6a9d-maas-api-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:19:12.345810 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:12.345616 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4246d01-d173-47fb-881a-dc2594ad6a9d-openshift-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:19:12.345810 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:12.345630 2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4246d01-d173-47fb-881a-dc2594ad6a9d-metrics-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:19:12.349253 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:12.349224 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk"] Sep 14 11:19:12.349497 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:12.349469 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" podUID="d634c091-026a-4f44-8ba6-5cd47b8e2d8a" containerName="istio-proxy" containerID="cri-o://7a210b98c9ead016c070e7400152e50bf414649fa02451f51008fd1aff493332" gracePeriod=30 Sep 14 11:19:13.195777 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:13.195737 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq" event={"ID":"d4246d01-d173-47fb-881a-dc2594ad6a9d","Type":"ContainerDied","Data":"d0b834152336d04fb00caad6194b58a8dcc825692ff67656535649ba924963e6"} Sep 14 11:19:13.196367 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:13.196346 2556 scope.go:117] "RemoveContainer" containerID="d3cff781a340c00491b095f9e7c97dbee23edd19b16b469d9b67671c62f69e39" Sep 14 11:19:13.196540 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:13.196008 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq" Sep 14 11:19:13.227159 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:13.227137 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq"] Sep 14 11:19:13.235324 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:13.235299 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6fcec1be-6f8c655c5c-g26vq"] Sep 14 11:19:14.382497 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:14.382447 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4246d01-d173-47fb-881a-dc2594ad6a9d" path="/var/lib/kubelet/pods/d4246d01-d173-47fb-881a-dc2594ad6a9d/volumes" Sep 14 11:19:16.008658 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:16.008627 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:19:18.893639 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:18.893615 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:19:19.003226 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.003164 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-workload-socket\") pod \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " Sep 14 11:19:19.003226 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.003194 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istiod-ca-cert\") pod \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " Sep 14 11:19:19.003226 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.003225 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istio-data\") pod \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " Sep 14 11:19:19.003454 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.003252 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istio-envoy\") pod \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " Sep 14 11:19:19.003454 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.003285 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7kkg\" (UniqueName: \"kubernetes.io/projected/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-kube-api-access-k7kkg\") pod \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " Sep 14 11:19:19.003454 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.003303 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istio-podinfo\") pod \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " Sep 14 11:19:19.003454 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.003319 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-credential-socket\") pod \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " Sep 14 11:19:19.003454 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.003352 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istio-token\") pod \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " Sep 14 11:19:19.003454 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.003375 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-workload-certs\") pod \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\" (UID: \"d634c091-026a-4f44-8ba6-5cd47b8e2d8a\") " Sep 14 11:19:19.003747 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.003447 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d634c091-026a-4f44-8ba6-5cd47b8e2d8a" (UID: "d634c091-026a-4f44-8ba6-5cd47b8e2d8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:19:19.003747 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.003591 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istio-data" (OuterVolumeSpecName: "istio-data") pod "d634c091-026a-4f44-8ba6-5cd47b8e2d8a" (UID: "d634c091-026a-4f44-8ba6-5cd47b8e2d8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:19:19.003747 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.003654 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d634c091-026a-4f44-8ba6-5cd47b8e2d8a" (UID: "d634c091-026a-4f44-8ba6-5cd47b8e2d8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:19:19.003747 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.003656 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d634c091-026a-4f44-8ba6-5cd47b8e2d8a" (UID: "d634c091-026a-4f44-8ba6-5cd47b8e2d8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:19:19.003905 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.003750 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:19:19.003905 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.003768 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:19:19.003905 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.003804 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d634c091-026a-4f44-8ba6-5cd47b8e2d8a" (UID: "d634c091-026a-4f44-8ba6-5cd47b8e2d8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:19:19.005453 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.005424 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-kube-api-access-k7kkg" (OuterVolumeSpecName: "kube-api-access-k7kkg") pod "d634c091-026a-4f44-8ba6-5cd47b8e2d8a" (UID: "d634c091-026a-4f44-8ba6-5cd47b8e2d8a"). InnerVolumeSpecName "kube-api-access-k7kkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:19:19.005566 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.005513 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d634c091-026a-4f44-8ba6-5cd47b8e2d8a" (UID: "d634c091-026a-4f44-8ba6-5cd47b8e2d8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 11:19:19.005734 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.005719 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d634c091-026a-4f44-8ba6-5cd47b8e2d8a" (UID: "d634c091-026a-4f44-8ba6-5cd47b8e2d8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:19:19.005991 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.005973 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istio-token" (OuterVolumeSpecName: "istio-token") pod "d634c091-026a-4f44-8ba6-5cd47b8e2d8a" (UID: "d634c091-026a-4f44-8ba6-5cd47b8e2d8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:19:19.104663 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.104633 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7kkg\" (UniqueName: \"kubernetes.io/projected/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-kube-api-access-k7kkg\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:19:19.104663 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.104660 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:19:19.104825 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.104669 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:19:19.104825 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.104678 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:19:19.104825 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.104686 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:19:19.104825 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.104693 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:19:19.104825 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.104700 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d634c091-026a-4f44-8ba6-5cd47b8e2d8a-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:19:19.234341 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.234311 2556 generic.go:356] "Generic (PLEG): container finished" podID="d634c091-026a-4f44-8ba6-5cd47b8e2d8a" containerID="7a210b98c9ead016c070e7400152e50bf414649fa02451f51008fd1aff493332" exitCode=0 Sep 14 11:19:19.234510 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.234349 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" event={"ID":"d634c091-026a-4f44-8ba6-5cd47b8e2d8a","Type":"ContainerDied","Data":"7a210b98c9ead016c070e7400152e50bf414649fa02451f51008fd1aff493332"} Sep 14 11:19:19.234510 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.234368 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" Sep 14 11:19:19.234510 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.234391 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk" event={"ID":"d634c091-026a-4f44-8ba6-5cd47b8e2d8a","Type":"ContainerDied","Data":"5e6bcbdebf66dc125d5dbaf252c1b400be44ab24c80846ee24c44c0e873900f1"} Sep 14 11:19:19.234510 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.234413 2556 scope.go:117] "RemoveContainer" containerID="7a210b98c9ead016c070e7400152e50bf414649fa02451f51008fd1aff493332" Sep 14 11:19:19.242925 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.242912 2556 scope.go:117] "RemoveContainer" containerID="928601cbb9768c18bde3bbc7f10c33d4d74a5eb213d640394f1e1ea3edc5c220" Sep 14 11:19:19.250270 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.250252 2556 scope.go:117] "RemoveContainer" containerID="7a210b98c9ead016c070e7400152e50bf414649fa02451f51008fd1aff493332" Sep 14 11:19:19.250530 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:19:19.250511 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a210b98c9ead016c070e7400152e50bf414649fa02451f51008fd1aff493332\": container with ID starting with 7a210b98c9ead016c070e7400152e50bf414649fa02451f51008fd1aff493332 not found: ID does not exist" containerID="7a210b98c9ead016c070e7400152e50bf414649fa02451f51008fd1aff493332" Sep 14 11:19:19.250575 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.250538 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a210b98c9ead016c070e7400152e50bf414649fa02451f51008fd1aff493332"} err="failed to get container status \"7a210b98c9ead016c070e7400152e50bf414649fa02451f51008fd1aff493332\": rpc error: code = NotFound desc = could not find container \"7a210b98c9ead016c070e7400152e50bf414649fa02451f51008fd1aff493332\": container with ID starting with 7a210b98c9ead016c070e7400152e50bf414649fa02451f51008fd1aff493332 not found: ID does not exist" Sep 14 11:19:19.250575 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.250556 2556 scope.go:117] "RemoveContainer" containerID="928601cbb9768c18bde3bbc7f10c33d4d74a5eb213d640394f1e1ea3edc5c220" Sep 14 11:19:19.250779 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:19:19.250764 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"928601cbb9768c18bde3bbc7f10c33d4d74a5eb213d640394f1e1ea3edc5c220\": container with ID starting with 928601cbb9768c18bde3bbc7f10c33d4d74a5eb213d640394f1e1ea3edc5c220 not found: ID does not exist" containerID="928601cbb9768c18bde3bbc7f10c33d4d74a5eb213d640394f1e1ea3edc5c220" Sep 14 11:19:19.250819 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.250786 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"928601cbb9768c18bde3bbc7f10c33d4d74a5eb213d640394f1e1ea3edc5c220"} err="failed to get container status \"928601cbb9768c18bde3bbc7f10c33d4d74a5eb213d640394f1e1ea3edc5c220\": rpc error: code = NotFound desc = could not find container \"928601cbb9768c18bde3bbc7f10c33d4d74a5eb213d640394f1e1ea3edc5c220\": container with ID starting with 928601cbb9768c18bde3bbc7f10c33d4d74a5eb213d640394f1e1ea3edc5c220 not found: ID does not exist" Sep 14 11:19:19.258473 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.258424 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk"] Sep 14 11:19:19.264821 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:19.264801 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-6fcec1be-openshift-default-5f7845b44d-5chxk"] Sep 14 11:19:20.389596 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:20.389559 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d634c091-026a-4f44-8ba6-5cd47b8e2d8a" path="/var/lib/kubelet/pods/d634c091-026a-4f44-8ba6-5cd47b8e2d8a/volumes" Sep 14 11:19:32.057730 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.057700 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2"] Sep 14 11:19:32.058161 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.058019 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4246d01-d173-47fb-881a-dc2594ad6a9d" containerName="maas-api" Sep 14 11:19:32.058161 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.058029 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4246d01-d173-47fb-881a-dc2594ad6a9d" containerName="maas-api" Sep 14 11:19:32.058161 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.058046 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d634c091-026a-4f44-8ba6-5cd47b8e2d8a" containerName="istio-proxy" Sep 14 11:19:32.058161 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.058051 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d634c091-026a-4f44-8ba6-5cd47b8e2d8a" containerName="istio-proxy" Sep 14 11:19:32.058161 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.058059 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d634c091-026a-4f44-8ba6-5cd47b8e2d8a" containerName="istio-proxy" Sep 14 11:19:32.058161 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.058064 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d634c091-026a-4f44-8ba6-5cd47b8e2d8a" containerName="istio-proxy" Sep 14 11:19:32.058161 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.058132 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d634c091-026a-4f44-8ba6-5cd47b8e2d8a" containerName="istio-proxy" Sep 14 11:19:32.058161 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.058141 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4246d01-d173-47fb-881a-dc2594ad6a9d" containerName="maas-api" Sep 14 11:19:32.058161 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.058147 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d634c091-026a-4f44-8ba6-5cd47b8e2d8a" containerName="istio-proxy" Sep 14 11:19:32.062538 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.062515 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:19:32.065762 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.065740 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6f3d0f\"/\"openshift-service-ca.crt\"" Sep 14 11:19:32.066942 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.066919 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6f3d0f\"/\"premium-w0-6f3d0f-kserve-self-signed-certs\"" Sep 14 11:19:32.067073 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.066959 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6f3d0f\"/\"kube-root-ca.crt\"" Sep 14 11:19:32.067204 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.067067 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6f3d0f\"/\"default-dockercfg-f2rbx\"" Sep 14 11:19:32.073032 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.073010 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2"] Sep 14 11:19:32.223623 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.223566 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-tmp-dir\") pod \"premium-w0-6f3d0f-kserve-684cd8899-ztgg2\" (UID: \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\") " pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:19:32.223795 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.223650 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-dshm\") pod \"premium-w0-6f3d0f-kserve-684cd8899-ztgg2\" (UID: \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\") " pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:19:32.223795 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.223687 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvx4q\" (UniqueName: \"kubernetes.io/projected/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-kube-api-access-jvx4q\") pod \"premium-w0-6f3d0f-kserve-684cd8899-ztgg2\" (UID: \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\") " pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:19:32.223795 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.223719 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-model-cache\") pod \"premium-w0-6f3d0f-kserve-684cd8899-ztgg2\" (UID: \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\") " pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:19:32.223795 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.223766 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-home\") pod \"premium-w0-6f3d0f-kserve-684cd8899-ztgg2\" (UID: \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\") " pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:19:32.224004 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.223854 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-tls-certs\") pod \"premium-w0-6f3d0f-kserve-684cd8899-ztgg2\" (UID: \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\") " pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:19:32.324960 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.324880 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-tmp-dir\") pod \"premium-w0-6f3d0f-kserve-684cd8899-ztgg2\" (UID: \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\") " pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:19:32.324960 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.324926 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-dshm\") pod \"premium-w0-6f3d0f-kserve-684cd8899-ztgg2\" (UID: \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\") " pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:19:32.325214 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.325035 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvx4q\" (UniqueName: \"kubernetes.io/projected/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-kube-api-access-jvx4q\") pod \"premium-w0-6f3d0f-kserve-684cd8899-ztgg2\" (UID: \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\") " pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:19:32.325214 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.325075 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-model-cache\") pod \"premium-w0-6f3d0f-kserve-684cd8899-ztgg2\" (UID: \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\") " pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:19:32.325214 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.325136 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-home\") pod \"premium-w0-6f3d0f-kserve-684cd8899-ztgg2\" (UID: \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\") " pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:19:32.325214 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.325208 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-tls-certs\") pod \"premium-w0-6f3d0f-kserve-684cd8899-ztgg2\" (UID: \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\") " pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:19:32.325459 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.325250 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-tmp-dir\") pod \"premium-w0-6f3d0f-kserve-684cd8899-ztgg2\" (UID: \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\") " pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:19:32.325459 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.325421 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-model-cache\") pod \"premium-w0-6f3d0f-kserve-684cd8899-ztgg2\" (UID: \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\") " pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:19:32.325529 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.325494 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-home\") pod \"premium-w0-6f3d0f-kserve-684cd8899-ztgg2\" (UID: \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\") " pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:19:32.327123 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.327102 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-dshm\") pod \"premium-w0-6f3d0f-kserve-684cd8899-ztgg2\" (UID: \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\") " pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:19:32.327426 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.327411 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-tls-certs\") pod \"premium-w0-6f3d0f-kserve-684cd8899-ztgg2\" (UID: \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\") " pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:19:32.341377 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.341352 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvx4q\" (UniqueName: \"kubernetes.io/projected/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-kube-api-access-jvx4q\") pod \"premium-w0-6f3d0f-kserve-684cd8899-ztgg2\" (UID: \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\") " pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:19:32.373452 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.373432 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:19:32.684705 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:32.684677 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2"] Sep 14 11:19:32.687394 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:19:32.687369 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4138f69_d4ca_4f25_ae4f_4e889bf8a972.slice/crio-8994dab17ad3639c93bb521a419ff890d6c047908b078b0058c4f3fd530e402c WatchSource:0}: Error finding container 8994dab17ad3639c93bb521a419ff890d6c047908b078b0058c4f3fd530e402c: Status 404 returned error can't find the container with id 8994dab17ad3639c93bb521a419ff890d6c047908b078b0058c4f3fd530e402c Sep 14 11:19:33.308466 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:33.308432 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" event={"ID":"f4138f69-d4ca-4f25-ae4f-4e889bf8a972","Type":"ContainerStarted","Data":"6d17f337dc1c494799da4929a8bf6a7dc95e9cc7c17f9898864c64372ca383f5"} Sep 14 11:19:33.308832 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:33.308476 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" event={"ID":"f4138f69-d4ca-4f25-ae4f-4e889bf8a972","Type":"ContainerStarted","Data":"8994dab17ad3639c93bb521a419ff890d6c047908b078b0058c4f3fd530e402c"} Sep 14 11:19:33.330186 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:33.330132 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" podStartSLOduration=1.3301116130000001 podStartE2EDuration="1.330111613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:19:33.327960908 +0000 UTC m=+1131.391363818" watchObservedRunningTime="2026-09-14 11:19:33.330111613 +0000 UTC m=+1131.393514526" Sep 14 11:19:42.373762 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:42.373734 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:19:42.374137 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:42.373773 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:19:42.388426 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:42.388404 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:19:43.368543 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:19:43.368517 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:20:05.438698 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.438630 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972"] Sep 14 11:20:05.443633 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.443616 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:20:05.456341 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.456319 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-39ab34\"/\"kube-root-ca.crt\"" Sep 14 11:20:05.456442 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.456379 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-39ab34\"/\"test-model-39ab34-kserve-self-signed-certs\"" Sep 14 11:20:05.456508 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.456492 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-39ab34\"/\"default-dockercfg-p6vxd\"" Sep 14 11:20:05.462324 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.462310 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-39ab34\"/\"openshift-service-ca.crt\"" Sep 14 11:20:05.477375 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.477351 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972"] Sep 14 11:20:05.489350 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.489323 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a21e8f-68d0-47ba-b030-81f39e58b460-model-cache\") pod \"test-model-39ab34-kserve-6b4c74959c-d7972\" (UID: \"f6a21e8f-68d0-47ba-b030-81f39e58b460\") " pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:20:05.489464 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.489442 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a21e8f-68d0-47ba-b030-81f39e58b460-dshm\") pod \"test-model-39ab34-kserve-6b4c74959c-d7972\" (UID: \"f6a21e8f-68d0-47ba-b030-81f39e58b460\") " pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:20:05.489526 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.489493 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6cxd\" (UniqueName: \"kubernetes.io/projected/f6a21e8f-68d0-47ba-b030-81f39e58b460-kube-api-access-w6cxd\") pod \"test-model-39ab34-kserve-6b4c74959c-d7972\" (UID: \"f6a21e8f-68d0-47ba-b030-81f39e58b460\") " pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:20:05.489595 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.489523 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a21e8f-68d0-47ba-b030-81f39e58b460-tls-certs\") pod \"test-model-39ab34-kserve-6b4c74959c-d7972\" (UID: \"f6a21e8f-68d0-47ba-b030-81f39e58b460\") " pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:20:05.489654 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.489613 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a21e8f-68d0-47ba-b030-81f39e58b460-tmp-dir\") pod \"test-model-39ab34-kserve-6b4c74959c-d7972\" (UID: \"f6a21e8f-68d0-47ba-b030-81f39e58b460\") " pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:20:05.489712 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.489674 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a21e8f-68d0-47ba-b030-81f39e58b460-home\") pod \"test-model-39ab34-kserve-6b4c74959c-d7972\" (UID: \"f6a21e8f-68d0-47ba-b030-81f39e58b460\") " pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:20:05.591127 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.591077 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a21e8f-68d0-47ba-b030-81f39e58b460-model-cache\") pod \"test-model-39ab34-kserve-6b4c74959c-d7972\" (UID: \"f6a21e8f-68d0-47ba-b030-81f39e58b460\") " pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:20:05.591291 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.591240 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a21e8f-68d0-47ba-b030-81f39e58b460-dshm\") pod \"test-model-39ab34-kserve-6b4c74959c-d7972\" (UID: \"f6a21e8f-68d0-47ba-b030-81f39e58b460\") " pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:20:05.591291 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.591269 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6cxd\" (UniqueName: \"kubernetes.io/projected/f6a21e8f-68d0-47ba-b030-81f39e58b460-kube-api-access-w6cxd\") pod \"test-model-39ab34-kserve-6b4c74959c-d7972\" (UID: \"f6a21e8f-68d0-47ba-b030-81f39e58b460\") " pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:20:05.591398 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.591290 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a21e8f-68d0-47ba-b030-81f39e58b460-tls-certs\") pod \"test-model-39ab34-kserve-6b4c74959c-d7972\" (UID: \"f6a21e8f-68d0-47ba-b030-81f39e58b460\") " pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:20:05.591398 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.591327 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a21e8f-68d0-47ba-b030-81f39e58b460-tmp-dir\") pod \"test-model-39ab34-kserve-6b4c74959c-d7972\" (UID: \"f6a21e8f-68d0-47ba-b030-81f39e58b460\") " pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:20:05.591398 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.591367 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a21e8f-68d0-47ba-b030-81f39e58b460-home\") pod \"test-model-39ab34-kserve-6b4c74959c-d7972\" (UID: \"f6a21e8f-68d0-47ba-b030-81f39e58b460\") " pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:20:05.591571 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.591543 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a21e8f-68d0-47ba-b030-81f39e58b460-model-cache\") pod \"test-model-39ab34-kserve-6b4c74959c-d7972\" (UID: \"f6a21e8f-68d0-47ba-b030-81f39e58b460\") " pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:20:05.591705 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.591685 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a21e8f-68d0-47ba-b030-81f39e58b460-home\") pod \"test-model-39ab34-kserve-6b4c74959c-d7972\" (UID: \"f6a21e8f-68d0-47ba-b030-81f39e58b460\") " pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:20:05.591758 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.591728 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a21e8f-68d0-47ba-b030-81f39e58b460-tmp-dir\") pod \"test-model-39ab34-kserve-6b4c74959c-d7972\" (UID: \"f6a21e8f-68d0-47ba-b030-81f39e58b460\") " pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:20:05.593532 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.593502 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a21e8f-68d0-47ba-b030-81f39e58b460-dshm\") pod \"test-model-39ab34-kserve-6b4c74959c-d7972\" (UID: \"f6a21e8f-68d0-47ba-b030-81f39e58b460\") " pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:20:05.593764 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.593745 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a21e8f-68d0-47ba-b030-81f39e58b460-tls-certs\") pod \"test-model-39ab34-kserve-6b4c74959c-d7972\" (UID: \"f6a21e8f-68d0-47ba-b030-81f39e58b460\") " pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:20:05.601737 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.601715 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6cxd\" (UniqueName: \"kubernetes.io/projected/f6a21e8f-68d0-47ba-b030-81f39e58b460-kube-api-access-w6cxd\") pod \"test-model-39ab34-kserve-6b4c74959c-d7972\" (UID: \"f6a21e8f-68d0-47ba-b030-81f39e58b460\") " pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:20:05.752448 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:05.752392 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:20:06.053750 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:06.053680 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972"] Sep 14 11:20:06.056267 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:20:06.056244 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6a21e8f_68d0_47ba_b030_81f39e58b460.slice/crio-0f4d3ea7635797f929d7f3c631a92477e9b6951bbe6314dfdf371eb911e7f982 WatchSource:0}: Error finding container 0f4d3ea7635797f929d7f3c631a92477e9b6951bbe6314dfdf371eb911e7f982: Status 404 returned error can't find the container with id 0f4d3ea7635797f929d7f3c631a92477e9b6951bbe6314dfdf371eb911e7f982 Sep 14 11:20:06.473670 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:06.473639 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" event={"ID":"f6a21e8f-68d0-47ba-b030-81f39e58b460","Type":"ContainerStarted","Data":"462c00bc7ee415f744d303a41929a32ea15a550d76c79d2c6aed6bd5aad587ee"} Sep 14 11:20:06.473670 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:06.473674 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" event={"ID":"f6a21e8f-68d0-47ba-b030-81f39e58b460","Type":"ContainerStarted","Data":"0f4d3ea7635797f929d7f3c631a92477e9b6951bbe6314dfdf371eb911e7f982"} Sep 14 11:20:15.752709 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:15.752679 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:20:15.753117 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:15.752912 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:20:15.764612 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:15.764589 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:20:15.794915 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:15.794865 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" podStartSLOduration=10.794850888 podStartE2EDuration="10.794850888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:20:06.500853658 +0000 UTC m=+1164.564256567" watchObservedRunningTime="2026-09-14 11:20:15.794850888 +0000 UTC m=+1173.858253797" Sep 14 11:20:16.531879 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:16.531850 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:20:47.668213 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:47.668181 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn"] Sep 14 11:20:47.668796 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:47.668390 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" podUID="743ca772-cec4-4dc3-9537-e0d2d9aa7987" containerName="maas-api" containerID="cri-o://bf81e1e8b90be7806edda4cd42ac15edae58292eb5e5e9cf9d32927a1f311f51" gracePeriod=30 Sep 14 11:20:47.901031 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:47.901006 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" Sep 14 11:20:47.965061 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:47.965001 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/743ca772-cec4-4dc3-9537-e0d2d9aa7987-metrics-certs\") pod \"743ca772-cec4-4dc3-9537-e0d2d9aa7987\" (UID: \"743ca772-cec4-4dc3-9537-e0d2d9aa7987\") " Sep 14 11:20:47.965061 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:47.965035 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpzp7\" (UniqueName: \"kubernetes.io/projected/743ca772-cec4-4dc3-9537-e0d2d9aa7987-kube-api-access-qpzp7\") pod \"743ca772-cec4-4dc3-9537-e0d2d9aa7987\" (UID: \"743ca772-cec4-4dc3-9537-e0d2d9aa7987\") " Sep 14 11:20:47.965226 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:47.965082 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/743ca772-cec4-4dc3-9537-e0d2d9aa7987-maas-api-tls\") pod \"743ca772-cec4-4dc3-9537-e0d2d9aa7987\" (UID: \"743ca772-cec4-4dc3-9537-e0d2d9aa7987\") " Sep 14 11:20:47.965226 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:47.965132 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/743ca772-cec4-4dc3-9537-e0d2d9aa7987-openshift-service-ca\") pod \"743ca772-cec4-4dc3-9537-e0d2d9aa7987\" (UID: \"743ca772-cec4-4dc3-9537-e0d2d9aa7987\") " Sep 14 11:20:47.965547 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:47.965514 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/743ca772-cec4-4dc3-9537-e0d2d9aa7987-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "743ca772-cec4-4dc3-9537-e0d2d9aa7987" (UID: "743ca772-cec4-4dc3-9537-e0d2d9aa7987"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:20:47.967111 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:47.967073 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/743ca772-cec4-4dc3-9537-e0d2d9aa7987-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "743ca772-cec4-4dc3-9537-e0d2d9aa7987" (UID: "743ca772-cec4-4dc3-9537-e0d2d9aa7987"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:20:47.967198 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:47.967123 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/743ca772-cec4-4dc3-9537-e0d2d9aa7987-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "743ca772-cec4-4dc3-9537-e0d2d9aa7987" (UID: "743ca772-cec4-4dc3-9537-e0d2d9aa7987"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:20:47.967198 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:47.967174 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/743ca772-cec4-4dc3-9537-e0d2d9aa7987-kube-api-access-qpzp7" (OuterVolumeSpecName: "kube-api-access-qpzp7") pod "743ca772-cec4-4dc3-9537-e0d2d9aa7987" (UID: "743ca772-cec4-4dc3-9537-e0d2d9aa7987"). InnerVolumeSpecName "kube-api-access-qpzp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:20:48.067073 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:48.067036 2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/743ca772-cec4-4dc3-9537-e0d2d9aa7987-metrics-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:20:48.067406 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:48.067378 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpzp7\" (UniqueName: \"kubernetes.io/projected/743ca772-cec4-4dc3-9537-e0d2d9aa7987-kube-api-access-qpzp7\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:20:48.067543 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:48.067410 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/743ca772-cec4-4dc3-9537-e0d2d9aa7987-maas-api-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:20:48.067543 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:48.067426 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/743ca772-cec4-4dc3-9537-e0d2d9aa7987-openshift-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:20:48.678875 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:48.678791 2556 generic.go:356] "Generic (PLEG): container finished" podID="743ca772-cec4-4dc3-9537-e0d2d9aa7987" containerID="bf81e1e8b90be7806edda4cd42ac15edae58292eb5e5e9cf9d32927a1f311f51" exitCode=0 Sep 14 11:20:48.678875 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:48.678837 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" Sep 14 11:20:48.679307 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:48.678877 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" event={"ID":"743ca772-cec4-4dc3-9537-e0d2d9aa7987","Type":"ContainerDied","Data":"bf81e1e8b90be7806edda4cd42ac15edae58292eb5e5e9cf9d32927a1f311f51"} Sep 14 11:20:48.679307 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:48.678934 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn" event={"ID":"743ca772-cec4-4dc3-9537-e0d2d9aa7987","Type":"ContainerDied","Data":"f076a3e7d357688dca253d3c5a98d2ba3d48b9027891eaca9451a5e16d52da3a"} Sep 14 11:20:48.679307 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:48.678960 2556 scope.go:117] "RemoveContainer" containerID="bf81e1e8b90be7806edda4cd42ac15edae58292eb5e5e9cf9d32927a1f311f51" Sep 14 11:20:48.691582 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:48.691511 2556 scope.go:117] "RemoveContainer" containerID="bf81e1e8b90be7806edda4cd42ac15edae58292eb5e5e9cf9d32927a1f311f51" Sep 14 11:20:48.691941 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:20:48.691920 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf81e1e8b90be7806edda4cd42ac15edae58292eb5e5e9cf9d32927a1f311f51\": container with ID starting with bf81e1e8b90be7806edda4cd42ac15edae58292eb5e5e9cf9d32927a1f311f51 not found: ID does not exist" containerID="bf81e1e8b90be7806edda4cd42ac15edae58292eb5e5e9cf9d32927a1f311f51" Sep 14 11:20:48.692032 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:48.691949 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf81e1e8b90be7806edda4cd42ac15edae58292eb5e5e9cf9d32927a1f311f51"} err="failed to get container status \"bf81e1e8b90be7806edda4cd42ac15edae58292eb5e5e9cf9d32927a1f311f51\": rpc error: code = NotFound desc = could not find container \"bf81e1e8b90be7806edda4cd42ac15edae58292eb5e5e9cf9d32927a1f311f51\": container with ID starting with bf81e1e8b90be7806edda4cd42ac15edae58292eb5e5e9cf9d32927a1f311f51 not found: ID does not exist" Sep 14 11:20:48.698725 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:48.698704 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn"] Sep 14 11:20:48.702924 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:48.702900 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-41158e-558f79fd7-vnzkn"] Sep 14 11:20:50.386691 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:50.386651 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="743ca772-cec4-4dc3-9537-e0d2d9aa7987" path="/var/lib/kubelet/pods/743ca772-cec4-4dc3-9537-e0d2d9aa7987/volumes" Sep 14 11:20:52.033167 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.033059 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l"] Sep 14 11:20:52.033510 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.033429 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="743ca772-cec4-4dc3-9537-e0d2d9aa7987" containerName="maas-api" Sep 14 11:20:52.033510 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.033442 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="743ca772-cec4-4dc3-9537-e0d2d9aa7987" containerName="maas-api" Sep 14 11:20:52.033510 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.033495 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="743ca772-cec4-4dc3-9537-e0d2d9aa7987" containerName="maas-api" Sep 14 11:20:52.038019 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.037994 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.040417 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.040397 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-508ed449-openshift-default-dockercfg-dmz2t\"" Sep 14 11:20:52.049918 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.049895 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l"] Sep 14 11:20:52.108005 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.107975 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-workload-socket\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.108164 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.108010 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-credential-socket\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.108164 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.108041 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-workload-certs\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.108164 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.108133 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f2c9a91-2957-4a86-a39d-3471d804193b-istio-podinfo\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.108279 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.108231 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c5rn\" (UniqueName: \"kubernetes.io/projected/3f2c9a91-2957-4a86-a39d-3471d804193b-kube-api-access-8c5rn\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.108279 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.108269 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-istio-envoy\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.108341 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.108285 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f2c9a91-2957-4a86-a39d-3471d804193b-istio-token\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.108341 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.108307 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-istio-data\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.108341 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.108327 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f2c9a91-2957-4a86-a39d-3471d804193b-istiod-ca-cert\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.209742 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.209712 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-workload-certs\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.209915 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.209747 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f2c9a91-2957-4a86-a39d-3471d804193b-istio-podinfo\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.209915 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.209825 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c5rn\" (UniqueName: \"kubernetes.io/projected/3f2c9a91-2957-4a86-a39d-3471d804193b-kube-api-access-8c5rn\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.209915 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.209857 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-istio-envoy\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.209915 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.209881 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f2c9a91-2957-4a86-a39d-3471d804193b-istio-token\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.209915 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.209912 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-istio-data\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.210203 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.209943 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f2c9a91-2957-4a86-a39d-3471d804193b-istiod-ca-cert\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.210203 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.210079 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-workload-socket\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.210203 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.210154 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-credential-socket\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.210344 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.210207 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-workload-certs\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.210391 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.210351 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-istio-data\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.210513 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.210493 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-credential-socket\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.210585 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.210573 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-workload-socket\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.210625 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.210595 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f2c9a91-2957-4a86-a39d-3471d804193b-istiod-ca-cert\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.212549 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.212530 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-istio-envoy\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.212745 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.212728 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f2c9a91-2957-4a86-a39d-3471d804193b-istio-podinfo\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.218432 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.218406 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f2c9a91-2957-4a86-a39d-3471d804193b-istio-token\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.218583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.218566 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c5rn\" (UniqueName: \"kubernetes.io/projected/3f2c9a91-2957-4a86-a39d-3471d804193b-kube-api-access-8c5rn\") pod \"e2e-mt-508ed449-openshift-default-76667d678d-wdz4l\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.347285 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.347209 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:52.629777 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.629752 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l"] Sep 14 11:20:52.631368 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:20:52.631341 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2c9a91_2957_4a86_a39d_3471d804193b.slice/crio-ef86684dbee43c93b3876b64af0d5f4b3d7d594125ceb481d82fda8c00f781a3 WatchSource:0}: Error finding container ef86684dbee43c93b3876b64af0d5f4b3d7d594125ceb481d82fda8c00f781a3: Status 404 returned error can't find the container with id ef86684dbee43c93b3876b64af0d5f4b3d7d594125ceb481d82fda8c00f781a3 Sep 14 11:20:52.633725 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.633684 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:20:52.633826 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.633769 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:20:52.633826 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.633811 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:20:52.701491 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.701466 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" event={"ID":"3f2c9a91-2957-4a86-a39d-3471d804193b","Type":"ContainerStarted","Data":"e5e2918e5e179858f7b98d2b7e9b6fcba794a1b3a9f3883baa1b6690c24b79f9"} Sep 14 11:20:52.701596 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.701510 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" event={"ID":"3f2c9a91-2957-4a86-a39d-3471d804193b","Type":"ContainerStarted","Data":"ef86684dbee43c93b3876b64af0d5f4b3d7d594125ceb481d82fda8c00f781a3"} Sep 14 11:20:52.730893 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:52.730836 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" podStartSLOduration=0.730817399 podStartE2EDuration="730.817399ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:20:52.728580069 +0000 UTC m=+1210.791982980" watchObservedRunningTime="2026-09-14 11:20:52.730817399 +0000 UTC m=+1210.794220312" Sep 14 11:20:53.347547 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:53.347520 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:53.352570 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:53.352546 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:53.706177 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:53.706145 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:20:53.706927 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:20:53.706909 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:21:31.543791 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.543759 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp"] Sep 14 11:21:31.554523 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.554499 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.556865 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.556846 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-8dd3f5-openshift-default-dockercfg-7vsg8\"" Sep 14 11:21:31.561191 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.561169 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp"] Sep 14 11:21:31.676702 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.676666 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-istio-envoy\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.678059 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.676719 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-credential-socket\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.678059 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.676748 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-istio-data\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.678059 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.676781 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69cks\" (UniqueName: \"kubernetes.io/projected/f72ec446-c7a5-4efb-b136-616064cf82f9-kube-api-access-69cks\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.678059 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.676800 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f72ec446-c7a5-4efb-b136-616064cf82f9-istio-podinfo\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.678059 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.676875 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-workload-socket\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.678059 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.676992 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f72ec446-c7a5-4efb-b136-616064cf82f9-istio-token\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.678059 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.677013 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-workload-certs\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.678059 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.677027 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f72ec446-c7a5-4efb-b136-616064cf82f9-istiod-ca-cert\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.778172 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.778137 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-credential-socket\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.778172 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.778172 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-istio-data\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.778395 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.778192 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69cks\" (UniqueName: \"kubernetes.io/projected/f72ec446-c7a5-4efb-b136-616064cf82f9-kube-api-access-69cks\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.778395 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.778208 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f72ec446-c7a5-4efb-b136-616064cf82f9-istio-podinfo\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.778395 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.778241 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-workload-socket\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.778395 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.778329 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f72ec446-c7a5-4efb-b136-616064cf82f9-istio-token\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.778395 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.778354 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-workload-certs\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.778395 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.778378 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f72ec446-c7a5-4efb-b136-616064cf82f9-istiod-ca-cert\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.778707 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.778426 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-istio-envoy\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.778707 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.778582 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-istio-data\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.778707 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.778625 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-credential-socket\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.778827 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.778736 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-workload-certs\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.778933 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.778912 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-workload-socket\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.779193 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.779174 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f72ec446-c7a5-4efb-b136-616064cf82f9-istiod-ca-cert\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.780545 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.780519 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-istio-envoy\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.780640 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.780620 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f72ec446-c7a5-4efb-b136-616064cf82f9-istio-podinfo\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.787143 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.787117 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f72ec446-c7a5-4efb-b136-616064cf82f9-istio-token\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.787335 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.787317 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69cks\" (UniqueName: \"kubernetes.io/projected/f72ec446-c7a5-4efb-b136-616064cf82f9-kube-api-access-69cks\") pod \"e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.866752 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.866725 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:31.899786 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.899755 2556 generic.go:356] "Generic (PLEG): container finished" podID="3f2c9a91-2957-4a86-a39d-3471d804193b" containerID="e5e2918e5e179858f7b98d2b7e9b6fcba794a1b3a9f3883baa1b6690c24b79f9" exitCode=0 Sep 14 11:21:31.899945 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.899834 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" event={"ID":"3f2c9a91-2957-4a86-a39d-3471d804193b","Type":"ContainerDied","Data":"e5e2918e5e179858f7b98d2b7e9b6fcba794a1b3a9f3883baa1b6690c24b79f9"} Sep 14 11:21:31.900282 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.900267 2556 scope.go:117] "RemoveContainer" containerID="e5e2918e5e179858f7b98d2b7e9b6fcba794a1b3a9f3883baa1b6690c24b79f9" Sep 14 11:21:31.901500 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.901465 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:21:31.901623 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.901550 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:21:31.901623 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:31.901598 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:21:32.142838 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:32.142754 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp"] Sep 14 11:21:32.145591 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:21:32.145562 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf72ec446_c7a5_4efb_b136_616064cf82f9.slice/crio-ef0d642140e8058bb32800c826e9e7c776dee58373a15762ee33bcb973acefca WatchSource:0}: Error finding container ef0d642140e8058bb32800c826e9e7c776dee58373a15762ee33bcb973acefca: Status 404 returned error can't find the container with id ef0d642140e8058bb32800c826e9e7c776dee58373a15762ee33bcb973acefca Sep 14 11:21:32.155218 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:32.148702 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:21:32.155218 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:32.148851 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:21:32.155218 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:32.148938 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:21:32.911320 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:32.911278 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" event={"ID":"3f2c9a91-2957-4a86-a39d-3471d804193b","Type":"ContainerStarted","Data":"0125dd23dc2312dc196b95838f4afc345b888d1f582bd447c29ed365ae65d20c"} Sep 14 11:21:32.914247 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:32.914219 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" event={"ID":"f72ec446-c7a5-4efb-b136-616064cf82f9","Type":"ContainerStarted","Data":"04a0767d569d44a024dbc387c0e0e1228a3d318b2429ccef11166ce5fad79d7b"} Sep 14 11:21:32.914411 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:32.914400 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" event={"ID":"f72ec446-c7a5-4efb-b136-616064cf82f9","Type":"ContainerStarted","Data":"ef0d642140e8058bb32800c826e9e7c776dee58373a15762ee33bcb973acefca"} Sep 14 11:21:32.960371 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:32.960321 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" podStartSLOduration=1.960303618 podStartE2EDuration="1.960303618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:21:32.958568982 +0000 UTC m=+1251.021971905" watchObservedRunningTime="2026-09-14 11:21:32.960303618 +0000 UTC m=+1251.023706527" Sep 14 11:21:33.347617 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:33.347532 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:21:33.347757 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:33.347729 2556 patch_prober.go:28] interesting pod/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 14 11:21:33.347803 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:33.347767 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" podUID="3f2c9a91-2957-4a86-a39d-3471d804193b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 14 11:21:33.867207 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:33.867168 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:33.872179 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:33.872154 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:33.919169 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:33.919137 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:33.919870 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:33.919854 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:21:34.348481 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:34.348399 2556 patch_prober.go:28] interesting pod/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 14 11:21:34.348481 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:34.348451 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" podUID="3f2c9a91-2957-4a86-a39d-3471d804193b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 14 11:21:35.348022 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:35.347942 2556 patch_prober.go:28] interesting pod/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 14 11:21:35.348022 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:35.347997 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" podUID="3f2c9a91-2957-4a86-a39d-3471d804193b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 14 11:21:36.347503 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:36.347470 2556 patch_prober.go:28] interesting pod/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 14 11:21:36.347683 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:36.347521 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" podUID="3f2c9a91-2957-4a86-a39d-3471d804193b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 14 11:21:37.348102 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:37.348037 2556 patch_prober.go:28] interesting pod/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 14 11:21:37.348567 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:37.348163 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" podUID="3f2c9a91-2957-4a86-a39d-3471d804193b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 14 11:21:38.352152 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:38.352120 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:21:38.352573 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:38.352407 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:21:38.353258 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:38.353237 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:21:53.961839 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:53.961801 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l"] Sep 14 11:21:53.962310 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:53.962101 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" podUID="3f2c9a91-2957-4a86-a39d-3471d804193b" containerName="istio-proxy" containerID="cri-o://0125dd23dc2312dc196b95838f4afc345b888d1f582bd447c29ed365ae65d20c" gracePeriod=30 Sep 14 11:21:59.216808 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.216783 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:21:59.350970 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.350899 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f2c9a91-2957-4a86-a39d-3471d804193b-istio-podinfo\") pod \"3f2c9a91-2957-4a86-a39d-3471d804193b\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " Sep 14 11:21:59.350970 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.350936 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f2c9a91-2957-4a86-a39d-3471d804193b-istiod-ca-cert\") pod \"3f2c9a91-2957-4a86-a39d-3471d804193b\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " Sep 14 11:21:59.351174 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.350982 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c5rn\" (UniqueName: \"kubernetes.io/projected/3f2c9a91-2957-4a86-a39d-3471d804193b-kube-api-access-8c5rn\") pod \"3f2c9a91-2957-4a86-a39d-3471d804193b\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " Sep 14 11:21:59.351174 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.351007 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f2c9a91-2957-4a86-a39d-3471d804193b-istio-token\") pod \"3f2c9a91-2957-4a86-a39d-3471d804193b\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " Sep 14 11:21:59.351174 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.351035 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-workload-socket\") pod \"3f2c9a91-2957-4a86-a39d-3471d804193b\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " Sep 14 11:21:59.351174 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.351135 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-istio-data\") pod \"3f2c9a91-2957-4a86-a39d-3471d804193b\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " Sep 14 11:21:59.351365 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.351223 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-credential-socket\") pod \"3f2c9a91-2957-4a86-a39d-3471d804193b\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " Sep 14 11:21:59.351365 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.351256 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-workload-certs\") pod \"3f2c9a91-2957-4a86-a39d-3471d804193b\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " Sep 14 11:21:59.351365 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.351289 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-istio-envoy\") pod \"3f2c9a91-2957-4a86-a39d-3471d804193b\" (UID: \"3f2c9a91-2957-4a86-a39d-3471d804193b\") " Sep 14 11:21:59.351365 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.351309 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3f2c9a91-2957-4a86-a39d-3471d804193b" (UID: "3f2c9a91-2957-4a86-a39d-3471d804193b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:21:59.351608 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.351405 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f2c9a91-2957-4a86-a39d-3471d804193b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3f2c9a91-2957-4a86-a39d-3471d804193b" (UID: "3f2c9a91-2957-4a86-a39d-3471d804193b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:21:59.351608 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.351439 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-istio-data" (OuterVolumeSpecName: "istio-data") pod "3f2c9a91-2957-4a86-a39d-3471d804193b" (UID: "3f2c9a91-2957-4a86-a39d-3471d804193b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:21:59.351608 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.351519 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3f2c9a91-2957-4a86-a39d-3471d804193b" (UID: "3f2c9a91-2957-4a86-a39d-3471d804193b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:21:59.351608 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.351577 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3f2c9a91-2957-4a86-a39d-3471d804193b" (UID: "3f2c9a91-2957-4a86-a39d-3471d804193b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:21:59.351814 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.351670 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:21:59.351814 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.351687 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:21:59.351814 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.351701 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f2c9a91-2957-4a86-a39d-3471d804193b-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:21:59.351814 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.351714 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:21:59.351814 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.351727 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:21:59.353304 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.353280 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3f2c9a91-2957-4a86-a39d-3471d804193b" (UID: "3f2c9a91-2957-4a86-a39d-3471d804193b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:21:59.353430 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.353330 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2c9a91-2957-4a86-a39d-3471d804193b-istio-token" (OuterVolumeSpecName: "istio-token") pod "3f2c9a91-2957-4a86-a39d-3471d804193b" (UID: "3f2c9a91-2957-4a86-a39d-3471d804193b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:21:59.353708 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.353433 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2c9a91-2957-4a86-a39d-3471d804193b-kube-api-access-8c5rn" (OuterVolumeSpecName: "kube-api-access-8c5rn") pod "3f2c9a91-2957-4a86-a39d-3471d804193b" (UID: "3f2c9a91-2957-4a86-a39d-3471d804193b"). InnerVolumeSpecName "kube-api-access-8c5rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:21:59.353708 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.353526 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f2c9a91-2957-4a86-a39d-3471d804193b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3f2c9a91-2957-4a86-a39d-3471d804193b" (UID: "3f2c9a91-2957-4a86-a39d-3471d804193b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 11:21:59.452334 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.452313 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f2c9a91-2957-4a86-a39d-3471d804193b-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:21:59.452334 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.452333 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f2c9a91-2957-4a86-a39d-3471d804193b-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:21:59.452462 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.452344 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c5rn\" (UniqueName: \"kubernetes.io/projected/3f2c9a91-2957-4a86-a39d-3471d804193b-kube-api-access-8c5rn\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:21:59.452462 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:21:59.452358 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f2c9a91-2957-4a86-a39d-3471d804193b-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:22:00.050602 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:00.050569 2556 generic.go:356] "Generic (PLEG): container finished" podID="3f2c9a91-2957-4a86-a39d-3471d804193b" containerID="0125dd23dc2312dc196b95838f4afc345b888d1f582bd447c29ed365ae65d20c" exitCode=0 Sep 14 11:22:00.050846 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:00.050625 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" Sep 14 11:22:00.050846 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:00.050654 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" event={"ID":"3f2c9a91-2957-4a86-a39d-3471d804193b","Type":"ContainerDied","Data":"0125dd23dc2312dc196b95838f4afc345b888d1f582bd447c29ed365ae65d20c"} Sep 14 11:22:00.050846 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:00.050706 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l" event={"ID":"3f2c9a91-2957-4a86-a39d-3471d804193b","Type":"ContainerDied","Data":"ef86684dbee43c93b3876b64af0d5f4b3d7d594125ceb481d82fda8c00f781a3"} Sep 14 11:22:00.050846 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:00.050742 2556 scope.go:117] "RemoveContainer" containerID="0125dd23dc2312dc196b95838f4afc345b888d1f582bd447c29ed365ae65d20c" Sep 14 11:22:00.059557 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:00.059538 2556 scope.go:117] "RemoveContainer" containerID="e5e2918e5e179858f7b98d2b7e9b6fcba794a1b3a9f3883baa1b6690c24b79f9" Sep 14 11:22:00.067026 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:00.067010 2556 scope.go:117] "RemoveContainer" containerID="0125dd23dc2312dc196b95838f4afc345b888d1f582bd447c29ed365ae65d20c" Sep 14 11:22:00.067287 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:22:00.067272 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0125dd23dc2312dc196b95838f4afc345b888d1f582bd447c29ed365ae65d20c\": container with ID starting with 0125dd23dc2312dc196b95838f4afc345b888d1f582bd447c29ed365ae65d20c not found: ID does not exist" containerID="0125dd23dc2312dc196b95838f4afc345b888d1f582bd447c29ed365ae65d20c" Sep 14 11:22:00.067346 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:00.067295 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0125dd23dc2312dc196b95838f4afc345b888d1f582bd447c29ed365ae65d20c"} err="failed to get container status \"0125dd23dc2312dc196b95838f4afc345b888d1f582bd447c29ed365ae65d20c\": rpc error: code = NotFound desc = could not find container \"0125dd23dc2312dc196b95838f4afc345b888d1f582bd447c29ed365ae65d20c\": container with ID starting with 0125dd23dc2312dc196b95838f4afc345b888d1f582bd447c29ed365ae65d20c not found: ID does not exist" Sep 14 11:22:00.067346 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:00.067310 2556 scope.go:117] "RemoveContainer" containerID="e5e2918e5e179858f7b98d2b7e9b6fcba794a1b3a9f3883baa1b6690c24b79f9" Sep 14 11:22:00.067525 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:22:00.067509 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5e2918e5e179858f7b98d2b7e9b6fcba794a1b3a9f3883baa1b6690c24b79f9\": container with ID starting with e5e2918e5e179858f7b98d2b7e9b6fcba794a1b3a9f3883baa1b6690c24b79f9 not found: ID does not exist" containerID="e5e2918e5e179858f7b98d2b7e9b6fcba794a1b3a9f3883baa1b6690c24b79f9" Sep 14 11:22:00.067565 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:00.067530 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5e2918e5e179858f7b98d2b7e9b6fcba794a1b3a9f3883baa1b6690c24b79f9"} err="failed to get container status \"e5e2918e5e179858f7b98d2b7e9b6fcba794a1b3a9f3883baa1b6690c24b79f9\": rpc error: code = NotFound desc = could not find container \"e5e2918e5e179858f7b98d2b7e9b6fcba794a1b3a9f3883baa1b6690c24b79f9\": container with ID starting with e5e2918e5e179858f7b98d2b7e9b6fcba794a1b3a9f3883baa1b6690c24b79f9 not found: ID does not exist" Sep 14 11:22:00.096488 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:00.096466 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l"] Sep 14 11:22:00.105639 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:00.105621 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-508ed449-openshift-default-76667d678d-wdz4l"] Sep 14 11:22:00.382402 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:00.382375 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f2c9a91-2957-4a86-a39d-3471d804193b" path="/var/lib/kubelet/pods/3f2c9a91-2957-4a86-a39d-3471d804193b/volumes" Sep 14 11:22:08.541603 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.541570 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252"] Sep 14 11:22:08.541959 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.541891 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f2c9a91-2957-4a86-a39d-3471d804193b" containerName="istio-proxy" Sep 14 11:22:08.541959 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.541902 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f2c9a91-2957-4a86-a39d-3471d804193b" containerName="istio-proxy" Sep 14 11:22:08.541959 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.541912 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f2c9a91-2957-4a86-a39d-3471d804193b" containerName="istio-proxy" Sep 14 11:22:08.541959 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.541917 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f2c9a91-2957-4a86-a39d-3471d804193b" containerName="istio-proxy" Sep 14 11:22:08.542119 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.541978 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f2c9a91-2957-4a86-a39d-3471d804193b" containerName="istio-proxy" Sep 14 11:22:08.542119 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.541988 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f2c9a91-2957-4a86-a39d-3471d804193b" containerName="istio-proxy" Sep 14 11:22:08.546561 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.546544 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.554183 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.554163 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-e42fe6-openshift-default-dockercfg-ztmcd\"" Sep 14 11:22:08.635471 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.635437 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252"] Sep 14 11:22:08.635951 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.635917 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-istio-envoy\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.636084 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.636037 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-workload-certs\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.636084 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.636065 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-istio-data\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.636218 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.636126 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s28bb\" (UniqueName: \"kubernetes.io/projected/1a14228d-a125-4d08-bf6c-f162f74f6330-kube-api-access-s28bb\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.636218 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.636175 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-workload-socket\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.636218 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.636190 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a14228d-a125-4d08-bf6c-f162f74f6330-istio-token\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.636328 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.636221 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a14228d-a125-4d08-bf6c-f162f74f6330-istiod-ca-cert\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.636328 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.636258 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-credential-socket\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.636328 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.636295 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a14228d-a125-4d08-bf6c-f162f74f6330-istio-podinfo\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.737467 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.737440 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-credential-socket\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.737634 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.737472 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a14228d-a125-4d08-bf6c-f162f74f6330-istio-podinfo\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.737634 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.737499 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-istio-envoy\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.737634 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.737570 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-workload-certs\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.737634 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.737594 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-istio-data\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.737634 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.737630 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s28bb\" (UniqueName: \"kubernetes.io/projected/1a14228d-a125-4d08-bf6c-f162f74f6330-kube-api-access-s28bb\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.737938 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.737681 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-workload-socket\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.737938 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.737707 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a14228d-a125-4d08-bf6c-f162f74f6330-istio-token\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.737938 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.737733 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a14228d-a125-4d08-bf6c-f162f74f6330-istiod-ca-cert\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.738113 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.737974 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-workload-certs\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.738113 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.738027 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-workload-socket\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.738227 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.738107 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-istio-data\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.738227 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.738165 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-credential-socket\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.738522 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.738498 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a14228d-a125-4d08-bf6c-f162f74f6330-istiod-ca-cert\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.739748 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.739727 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-istio-envoy\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.739953 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.739936 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a14228d-a125-4d08-bf6c-f162f74f6330-istio-podinfo\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.756952 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.756920 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s28bb\" (UniqueName: \"kubernetes.io/projected/1a14228d-a125-4d08-bf6c-f162f74f6330-kube-api-access-s28bb\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.771074 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.771044 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a14228d-a125-4d08-bf6c-f162f74f6330-istio-token\") pod \"e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:08.856403 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:08.856332 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:09.135836 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:09.135812 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252"] Sep 14 11:22:09.138352 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:22:09.138321 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a14228d_a125_4d08_bf6c_f162f74f6330.slice/crio-d7e859c169dc55f25ffa074a468cb15f6c52aae0e4db80b528253a98dd4cf86e WatchSource:0}: Error finding container d7e859c169dc55f25ffa074a468cb15f6c52aae0e4db80b528253a98dd4cf86e: Status 404 returned error can't find the container with id d7e859c169dc55f25ffa074a468cb15f6c52aae0e4db80b528253a98dd4cf86e Sep 14 11:22:09.143412 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:09.142013 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:22:09.144136 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:09.143646 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:22:09.144136 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:09.143759 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:22:10.102391 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:10.102359 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" event={"ID":"1a14228d-a125-4d08-bf6c-f162f74f6330","Type":"ContainerStarted","Data":"b09ca90c93278c463ecfc08aabd02bb45c09483cd95720b3795b20ea1ebf594a"} Sep 14 11:22:10.102874 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:10.102452 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" event={"ID":"1a14228d-a125-4d08-bf6c-f162f74f6330","Type":"ContainerStarted","Data":"d7e859c169dc55f25ffa074a468cb15f6c52aae0e4db80b528253a98dd4cf86e"} Sep 14 11:22:10.130844 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:10.130800 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" podStartSLOduration=2.130786219 podStartE2EDuration="2.130786219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:22:10.128040881 +0000 UTC m=+1288.191443791" watchObservedRunningTime="2026-09-14 11:22:10.130786219 +0000 UTC m=+1288.194189129" Sep 14 11:22:10.857459 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:10.857425 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:10.862603 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:10.862577 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:11.107457 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:11.107425 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:11.108531 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:11.108486 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:35.412499 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:35.412470 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp"] Sep 14 11:22:35.413051 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:35.412695 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" podUID="f72ec446-c7a5-4efb-b136-616064cf82f9" containerName="istio-proxy" containerID="cri-o://04a0767d569d44a024dbc387c0e0e1228a3d318b2429ccef11166ce5fad79d7b" gracePeriod=30 Sep 14 11:22:40.679469 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.679415 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:22:40.826635 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.826608 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69cks\" (UniqueName: \"kubernetes.io/projected/f72ec446-c7a5-4efb-b136-616064cf82f9-kube-api-access-69cks\") pod \"f72ec446-c7a5-4efb-b136-616064cf82f9\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " Sep 14 11:22:40.826797 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.826646 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-istio-data\") pod \"f72ec446-c7a5-4efb-b136-616064cf82f9\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " Sep 14 11:22:40.826797 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.826777 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-credential-socket\") pod \"f72ec446-c7a5-4efb-b136-616064cf82f9\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " Sep 14 11:22:40.826884 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.826831 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-workload-socket\") pod \"f72ec446-c7a5-4efb-b136-616064cf82f9\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " Sep 14 11:22:40.826884 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.826863 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f72ec446-c7a5-4efb-b136-616064cf82f9-istio-podinfo\") pod \"f72ec446-c7a5-4efb-b136-616064cf82f9\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " Sep 14 11:22:40.826986 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.826909 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f72ec446-c7a5-4efb-b136-616064cf82f9-istio-token\") pod \"f72ec446-c7a5-4efb-b136-616064cf82f9\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " Sep 14 11:22:40.826986 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.826935 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f72ec446-c7a5-4efb-b136-616064cf82f9-istiod-ca-cert\") pod \"f72ec446-c7a5-4efb-b136-616064cf82f9\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " Sep 14 11:22:40.826986 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.826962 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-istio-envoy\") pod \"f72ec446-c7a5-4efb-b136-616064cf82f9\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " Sep 14 11:22:40.827171 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.826933 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-istio-data" (OuterVolumeSpecName: "istio-data") pod "f72ec446-c7a5-4efb-b136-616064cf82f9" (UID: "f72ec446-c7a5-4efb-b136-616064cf82f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:22:40.827171 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.827033 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-workload-certs\") pod \"f72ec446-c7a5-4efb-b136-616064cf82f9\" (UID: \"f72ec446-c7a5-4efb-b136-616064cf82f9\") " Sep 14 11:22:40.827487 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.826995 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f72ec446-c7a5-4efb-b136-616064cf82f9" (UID: "f72ec446-c7a5-4efb-b136-616064cf82f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:22:40.827487 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.827134 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f72ec446-c7a5-4efb-b136-616064cf82f9" (UID: "f72ec446-c7a5-4efb-b136-616064cf82f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:22:40.827487 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.827359 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72ec446-c7a5-4efb-b136-616064cf82f9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f72ec446-c7a5-4efb-b136-616064cf82f9" (UID: "f72ec446-c7a5-4efb-b136-616064cf82f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:22:40.827487 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.827367 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f72ec446-c7a5-4efb-b136-616064cf82f9" (UID: "f72ec446-c7a5-4efb-b136-616064cf82f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:22:40.827487 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.827449 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:22:40.828882 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.828855 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72ec446-c7a5-4efb-b136-616064cf82f9-kube-api-access-69cks" (OuterVolumeSpecName: "kube-api-access-69cks") pod "f72ec446-c7a5-4efb-b136-616064cf82f9" (UID: "f72ec446-c7a5-4efb-b136-616064cf82f9"). InnerVolumeSpecName "kube-api-access-69cks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:22:40.828976 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.828958 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f72ec446-c7a5-4efb-b136-616064cf82f9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f72ec446-c7a5-4efb-b136-616064cf82f9" (UID: "f72ec446-c7a5-4efb-b136-616064cf82f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 11:22:40.829239 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.829215 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72ec446-c7a5-4efb-b136-616064cf82f9-istio-token" (OuterVolumeSpecName: "istio-token") pod "f72ec446-c7a5-4efb-b136-616064cf82f9" (UID: "f72ec446-c7a5-4efb-b136-616064cf82f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:22:40.829307 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.829244 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f72ec446-c7a5-4efb-b136-616064cf82f9" (UID: "f72ec446-c7a5-4efb-b136-616064cf82f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:22:40.928587 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.928558 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:22:40.928587 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.928583 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f72ec446-c7a5-4efb-b136-616064cf82f9-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:22:40.928587 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.928593 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f72ec446-c7a5-4efb-b136-616064cf82f9-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:22:40.928788 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.928602 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f72ec446-c7a5-4efb-b136-616064cf82f9-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:22:40.928788 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.928610 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:22:40.928788 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.928618 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:22:40.928788 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.928625 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69cks\" (UniqueName: \"kubernetes.io/projected/f72ec446-c7a5-4efb-b136-616064cf82f9-kube-api-access-69cks\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:22:40.928788 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:40.928634 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f72ec446-c7a5-4efb-b136-616064cf82f9-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:22:41.256935 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:41.256898 2556 generic.go:356] "Generic (PLEG): container finished" podID="f72ec446-c7a5-4efb-b136-616064cf82f9" containerID="04a0767d569d44a024dbc387c0e0e1228a3d318b2429ccef11166ce5fad79d7b" exitCode=0 Sep 14 11:22:41.257122 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:41.256947 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" Sep 14 11:22:41.257122 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:41.256977 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" event={"ID":"f72ec446-c7a5-4efb-b136-616064cf82f9","Type":"ContainerDied","Data":"04a0767d569d44a024dbc387c0e0e1228a3d318b2429ccef11166ce5fad79d7b"} Sep 14 11:22:41.257122 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:41.257035 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp" event={"ID":"f72ec446-c7a5-4efb-b136-616064cf82f9","Type":"ContainerDied","Data":"ef0d642140e8058bb32800c826e9e7c776dee58373a15762ee33bcb973acefca"} Sep 14 11:22:41.257122 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:41.257058 2556 scope.go:117] "RemoveContainer" containerID="04a0767d569d44a024dbc387c0e0e1228a3d318b2429ccef11166ce5fad79d7b" Sep 14 11:22:41.266097 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:41.266067 2556 scope.go:117] "RemoveContainer" containerID="04a0767d569d44a024dbc387c0e0e1228a3d318b2429ccef11166ce5fad79d7b" Sep 14 11:22:41.266374 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:22:41.266357 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04a0767d569d44a024dbc387c0e0e1228a3d318b2429ccef11166ce5fad79d7b\": container with ID starting with 04a0767d569d44a024dbc387c0e0e1228a3d318b2429ccef11166ce5fad79d7b not found: ID does not exist" containerID="04a0767d569d44a024dbc387c0e0e1228a3d318b2429ccef11166ce5fad79d7b" Sep 14 11:22:41.266448 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:41.266386 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04a0767d569d44a024dbc387c0e0e1228a3d318b2429ccef11166ce5fad79d7b"} err="failed to get container status \"04a0767d569d44a024dbc387c0e0e1228a3d318b2429ccef11166ce5fad79d7b\": rpc error: code = NotFound desc = could not find container \"04a0767d569d44a024dbc387c0e0e1228a3d318b2429ccef11166ce5fad79d7b\": container with ID starting with 04a0767d569d44a024dbc387c0e0e1228a3d318b2429ccef11166ce5fad79d7b not found: ID does not exist" Sep 14 11:22:41.293039 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:41.293011 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp"] Sep 14 11:22:41.296451 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:41.296424 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-8dd3f5-openshift-default-97ff569fb-jw4vp"] Sep 14 11:22:42.382417 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.382373 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f72ec446-c7a5-4efb-b136-616064cf82f9" path="/var/lib/kubelet/pods/f72ec446-c7a5-4efb-b136-616064cf82f9/volumes" Sep 14 11:22:42.747365 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.747281 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj"] Sep 14 11:22:42.747668 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.747655 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f72ec446-c7a5-4efb-b136-616064cf82f9" containerName="istio-proxy" Sep 14 11:22:42.747711 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.747670 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f72ec446-c7a5-4efb-b136-616064cf82f9" containerName="istio-proxy" Sep 14 11:22:42.747746 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.747737 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f72ec446-c7a5-4efb-b136-616064cf82f9" containerName="istio-proxy" Sep 14 11:22:42.752065 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.752050 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.758893 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.758875 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-41ae25-openshift-default-dockercfg-hrdsl\"" Sep 14 11:22:42.770252 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.770232 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj"] Sep 14 11:22:42.845630 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.845588 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istio-token\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.845932 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.845880 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-workload-certs\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.846156 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.846135 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-credential-socket\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.846321 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.846306 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istiod-ca-cert\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.846468 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.846443 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgsqg\" (UniqueName: \"kubernetes.io/projected/51fd3d77-7af2-4683-8b6b-f6783b654cb4-kube-api-access-mgsqg\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.846602 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.846551 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-workload-socket\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.846673 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.846604 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istio-envoy\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.846734 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.846677 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istio-podinfo\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.846734 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.846714 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istio-data\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.947755 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.947723 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istio-envoy\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.947912 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.947783 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istio-podinfo\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.947912 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.947814 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istio-data\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.947912 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.947834 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istio-token\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.947912 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.947869 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-workload-certs\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.947912 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.947896 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-credential-socket\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.948221 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.947941 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istiod-ca-cert\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.948221 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.947965 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgsqg\" (UniqueName: \"kubernetes.io/projected/51fd3d77-7af2-4683-8b6b-f6783b654cb4-kube-api-access-mgsqg\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.948221 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.948025 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-workload-socket\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.948378 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.948260 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istio-data\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.948675 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.948650 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-credential-socket\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.948796 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.948727 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-workload-certs\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.948796 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.948771 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-workload-socket\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.949531 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.949508 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istiod-ca-cert\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.950033 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.950012 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istio-envoy\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.950277 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.950259 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istio-podinfo\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.957419 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.957393 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istio-token\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:42.957652 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:42.957637 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgsqg\" (UniqueName: \"kubernetes.io/projected/51fd3d77-7af2-4683-8b6b-f6783b654cb4-kube-api-access-mgsqg\") pod \"e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:43.062328 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:43.062259 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:43.366140 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:43.365478 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj"] Sep 14 11:22:43.376647 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:43.376611 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:22:43.376733 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:43.376679 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:22:43.376733 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:43.376708 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:22:44.273815 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:44.273780 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" event={"ID":"51fd3d77-7af2-4683-8b6b-f6783b654cb4","Type":"ContainerStarted","Data":"002c204f76ea40de1a13d2f46433527665a4dbbe4ddd811b3fd87fce6b2aea11"} Sep 14 11:22:44.274293 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:44.273829 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" event={"ID":"51fd3d77-7af2-4683-8b6b-f6783b654cb4","Type":"ContainerStarted","Data":"622a55dd3dd665c5eafb104b7cbaa8901f55042dde103296fa6a94aff24d8259"} Sep 14 11:22:44.308597 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:44.308543 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" podStartSLOduration=2.3085279229999998 podStartE2EDuration="2.308527923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:22:44.306579914 +0000 UTC m=+1322.369982827" watchObservedRunningTime="2026-09-14 11:22:44.308527923 +0000 UTC m=+1322.371930833" Sep 14 11:22:45.062844 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:45.062808 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:45.067672 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:45.067651 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:45.279652 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:45.279624 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:45.280657 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:45.280637 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:22:46.285884 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:46.285784 2556 generic.go:356] "Generic (PLEG): container finished" podID="1a14228d-a125-4d08-bf6c-f162f74f6330" containerID="b09ca90c93278c463ecfc08aabd02bb45c09483cd95720b3795b20ea1ebf594a" exitCode=0 Sep 14 11:22:46.285884 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:46.285858 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" event={"ID":"1a14228d-a125-4d08-bf6c-f162f74f6330","Type":"ContainerDied","Data":"b09ca90c93278c463ecfc08aabd02bb45c09483cd95720b3795b20ea1ebf594a"} Sep 14 11:22:46.286423 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:46.286409 2556 scope.go:117] "RemoveContainer" containerID="b09ca90c93278c463ecfc08aabd02bb45c09483cd95720b3795b20ea1ebf594a" Sep 14 11:22:46.287476 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:46.287439 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:22:46.287571 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:46.287512 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:22:46.287571 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:46.287549 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:22:47.297432 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:47.297390 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" event={"ID":"1a14228d-a125-4d08-bf6c-f162f74f6330","Type":"ContainerStarted","Data":"4a7f621a38fe930a7a425016f9be52aa6c83c1342f1218e21d04c3fa15f31caa"} Sep 14 11:22:47.856956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:47.856921 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:47.857161 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:47.857138 2556 patch_prober.go:28] interesting pod/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 14 11:22:47.857227 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:47.857204 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" podUID="1a14228d-a125-4d08-bf6c-f162f74f6330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 14 11:22:48.857626 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:48.857593 2556 patch_prober.go:28] interesting pod/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 14 11:22:48.857994 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:48.857651 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" podUID="1a14228d-a125-4d08-bf6c-f162f74f6330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 14 11:22:49.857001 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:49.856968 2556 patch_prober.go:28] interesting pod/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 14 11:22:49.857173 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:49.857023 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" podUID="1a14228d-a125-4d08-bf6c-f162f74f6330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 14 11:22:50.857369 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:50.857337 2556 patch_prober.go:28] interesting pod/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 14 11:22:50.857801 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:50.857399 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" podUID="1a14228d-a125-4d08-bf6c-f162f74f6330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 14 11:22:51.856957 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:51.856916 2556 patch_prober.go:28] interesting pod/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 14 11:22:51.857157 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:51.856974 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" podUID="1a14228d-a125-4d08-bf6c-f162f74f6330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 14 11:22:52.860964 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:52.860936 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:52.861418 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:52.861291 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:22:52.861974 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:22:52.861956 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:23:03.179510 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:03.179475 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp"] Sep 14 11:23:13.439351 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:13.439296 2556 generic.go:356] "Generic (PLEG): container finished" podID="51fd3d77-7af2-4683-8b6b-f6783b654cb4" containerID="002c204f76ea40de1a13d2f46433527665a4dbbe4ddd811b3fd87fce6b2aea11" exitCode=0 Sep 14 11:23:13.439667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:13.439367 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" event={"ID":"51fd3d77-7af2-4683-8b6b-f6783b654cb4","Type":"ContainerDied","Data":"002c204f76ea40de1a13d2f46433527665a4dbbe4ddd811b3fd87fce6b2aea11"} Sep 14 11:23:13.439739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:13.439725 2556 scope.go:117] "RemoveContainer" containerID="002c204f76ea40de1a13d2f46433527665a4dbbe4ddd811b3fd87fce6b2aea11" Sep 14 11:23:13.440902 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:13.440877 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:23:13.440959 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:13.440943 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:23:13.440994 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:13.440973 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:23:14.446468 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:14.446435 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" event={"ID":"51fd3d77-7af2-4683-8b6b-f6783b654cb4","Type":"ContainerStarted","Data":"2e447df1dbed19d0b223d6e5f235e66edcd7f7afb8d349beeb726c21c0ae92ee"} Sep 14 11:23:15.062627 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:15.062589 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:23:15.063666 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:15.063637 2556 patch_prober.go:28] interesting pod/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 14 11:23:15.063790 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:15.063703 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" podUID="51fd3d77-7af2-4683-8b6b-f6783b654cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 14 11:23:15.280194 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:15.280150 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:23:16.062985 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:16.062946 2556 patch_prober.go:28] interesting pod/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 14 11:23:16.063490 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:16.063014 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" podUID="51fd3d77-7af2-4683-8b6b-f6783b654cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 14 11:23:16.981325 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:16.981289 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7935-58a94d"] Sep 14 11:23:16.983978 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:16.983949 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7935-58a94d" Sep 14 11:23:16.987891 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:16.987862 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5kv55\"" Sep 14 11:23:16.990628 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:16.990600 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6d272c-krztg"] Sep 14 11:23:16.997293 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:16.997271 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7935-58a94d"] Sep 14 11:23:16.997425 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:16.997391 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6d272c-krztg" Sep 14 11:23:16.999990 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:16.999964 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 11:23:17.000341 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:17.000317 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f98vt\"" Sep 14 11:23:17.027206 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:17.027162 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6d272c-krztg"] Sep 14 11:23:17.051278 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:17.051245 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmphv\" (UniqueName: \"kubernetes.io/projected/e5ab06f8-f3d7-40aa-b02e-14b193d0d5b3-kube-api-access-fmphv\") pod \"test-curl-7935-58a94d\" (UID: \"e5ab06f8-f3d7-40aa-b02e-14b193d0d5b3\") " pod="opendatahub/test-curl-7935-58a94d" Sep 14 11:23:17.051278 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:17.051283 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/986bc775-2278-492e-a152-25b8f4f12a5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6d272c-krztg\" (UID: \"986bc775-2278-492e-a152-25b8f4f12a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6d272c-krztg" Sep 14 11:23:17.062786 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:17.062757 2556 patch_prober.go:28] interesting pod/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 14 11:23:17.062888 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:17.062800 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" podUID="51fd3d77-7af2-4683-8b6b-f6783b654cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 14 11:23:17.152602 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:17.152560 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmphv\" (UniqueName: \"kubernetes.io/projected/e5ab06f8-f3d7-40aa-b02e-14b193d0d5b3-kube-api-access-fmphv\") pod \"test-curl-7935-58a94d\" (UID: \"e5ab06f8-f3d7-40aa-b02e-14b193d0d5b3\") " pod="opendatahub/test-curl-7935-58a94d" Sep 14 11:23:17.152602 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:17.152603 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/986bc775-2278-492e-a152-25b8f4f12a5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6d272c-krztg\" (UID: \"986bc775-2278-492e-a152-25b8f4f12a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6d272c-krztg" Sep 14 11:23:17.153352 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:17.153329 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/986bc775-2278-492e-a152-25b8f4f12a5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6d272c-krztg\" (UID: \"986bc775-2278-492e-a152-25b8f4f12a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6d272c-krztg" Sep 14 11:23:17.160816 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:17.160792 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmphv\" (UniqueName: \"kubernetes.io/projected/e5ab06f8-f3d7-40aa-b02e-14b193d0d5b3-kube-api-access-fmphv\") pod \"test-curl-7935-58a94d\" (UID: \"e5ab06f8-f3d7-40aa-b02e-14b193d0d5b3\") " pod="opendatahub/test-curl-7935-58a94d" Sep 14 11:23:17.297511 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:17.297405 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7935-58a94d" Sep 14 11:23:17.330421 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:17.330395 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6d272c-krztg" Sep 14 11:23:17.576938 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:17.576912 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7935-58a94d"] Sep 14 11:23:17.578542 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:23:17.578504 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5ab06f8_f3d7_40aa_b02e_14b193d0d5b3.slice/crio-d90c8a9c385d03e6034e92a46e34e69d5186a09757371edb971a5e2a852b984e WatchSource:0}: Error finding container d90c8a9c385d03e6034e92a46e34e69d5186a09757371edb971a5e2a852b984e: Status 404 returned error can't find the container with id d90c8a9c385d03e6034e92a46e34e69d5186a09757371edb971a5e2a852b984e Sep 14 11:23:17.580471 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:17.580456 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 11:23:17.635083 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:17.635058 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6d272c-krztg"] Sep 14 11:23:17.636765 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:23:17.636739 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod986bc775_2278_492e_a152_25b8f4f12a5c.slice/crio-ea0abacbdbe6596f0ade01f27905405ed68782a0737c570a271b9cf336ea55cf WatchSource:0}: Error finding container ea0abacbdbe6596f0ade01f27905405ed68782a0737c570a271b9cf336ea55cf: Status 404 returned error can't find the container with id ea0abacbdbe6596f0ade01f27905405ed68782a0737c570a271b9cf336ea55cf Sep 14 11:23:18.062980 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.062945 2556 patch_prober.go:28] interesting pod/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 14 11:23:18.063191 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.063014 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" podUID="51fd3d77-7af2-4683-8b6b-f6783b654cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 14 11:23:18.200858 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.200613 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" podUID="994ef2fb-3939-4282-a7b3-e08247dbf4f0" containerName="main" containerID="cri-o://5f0ba376d65e91d8b247f3e743a1eaa46cf72f512f724b1bc0ff4aa9140aa04c" gracePeriod=45 Sep 14 11:23:18.468417 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.468392 2556 generic.go:356] "Generic (PLEG): container finished" podID="994ef2fb-3939-4282-a7b3-e08247dbf4f0" containerID="5f0ba376d65e91d8b247f3e743a1eaa46cf72f512f724b1bc0ff4aa9140aa04c" exitCode=0 Sep 14 11:23:18.468535 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.468471 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" event={"ID":"994ef2fb-3939-4282-a7b3-e08247dbf4f0","Type":"ContainerDied","Data":"5f0ba376d65e91d8b247f3e743a1eaa46cf72f512f724b1bc0ff4aa9140aa04c"} Sep 14 11:23:18.468535 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.468523 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" event={"ID":"994ef2fb-3939-4282-a7b3-e08247dbf4f0","Type":"ContainerDied","Data":"5110460f12b087c92ee35ac3c48fe4d959ca436c2d952a0afbb341a9bbb55ee4"} Sep 14 11:23:18.468625 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.468542 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5110460f12b087c92ee35ac3c48fe4d959ca436c2d952a0afbb341a9bbb55ee4" Sep 14 11:23:18.469674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.469654 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7935-58a94d" event={"ID":"e5ab06f8-f3d7-40aa-b02e-14b193d0d5b3","Type":"ContainerStarted","Data":"d90c8a9c385d03e6034e92a46e34e69d5186a09757371edb971a5e2a852b984e"} Sep 14 11:23:18.474057 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.474035 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6d272c-krztg" event={"ID":"986bc775-2278-492e-a152-25b8f4f12a5c","Type":"ContainerStarted","Data":"732e083df2d1cb08cef3349fad1739accf68bad8b48f60a26e5185aa725c8dd9"} Sep 14 11:23:18.474188 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.474079 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6d272c-krztg" event={"ID":"986bc775-2278-492e-a152-25b8f4f12a5c","Type":"ContainerStarted","Data":"ea0abacbdbe6596f0ade01f27905405ed68782a0737c570a271b9cf336ea55cf"} Sep 14 11:23:18.477962 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.477944 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:23:18.520222 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.520179 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6d272c-krztg" podStartSLOduration=2.520163131 podStartE2EDuration="2.520163131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:23:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:23:18.517299571 +0000 UTC m=+1356.580702481" watchObservedRunningTime="2026-09-14 11:23:18.520163131 +0000 UTC m=+1356.583566061" Sep 14 11:23:18.567069 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.567046 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rs7hj\" (UniqueName: \"kubernetes.io/projected/994ef2fb-3939-4282-a7b3-e08247dbf4f0-kube-api-access-rs7hj\") pod \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\" (UID: \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\") " Sep 14 11:23:18.567194 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.567116 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/994ef2fb-3939-4282-a7b3-e08247dbf4f0-dshm\") pod \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\" (UID: \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\") " Sep 14 11:23:18.567194 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.567137 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/994ef2fb-3939-4282-a7b3-e08247dbf4f0-model-cache\") pod \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\" (UID: \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\") " Sep 14 11:23:18.567194 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.567179 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/994ef2fb-3939-4282-a7b3-e08247dbf4f0-home\") pod \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\" (UID: \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\") " Sep 14 11:23:18.567316 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.567222 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/994ef2fb-3939-4282-a7b3-e08247dbf4f0-tmp-dir\") pod \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\" (UID: \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\") " Sep 14 11:23:18.567367 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.567319 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/994ef2fb-3939-4282-a7b3-e08247dbf4f0-tls-certs\") pod \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\" (UID: \"994ef2fb-3939-4282-a7b3-e08247dbf4f0\") " Sep 14 11:23:18.567418 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.567356 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/994ef2fb-3939-4282-a7b3-e08247dbf4f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "994ef2fb-3939-4282-a7b3-e08247dbf4f0" (UID: "994ef2fb-3939-4282-a7b3-e08247dbf4f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:23:18.567474 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.567418 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/994ef2fb-3939-4282-a7b3-e08247dbf4f0-home" (OuterVolumeSpecName: "home") pod "994ef2fb-3939-4282-a7b3-e08247dbf4f0" (UID: "994ef2fb-3939-4282-a7b3-e08247dbf4f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:23:18.567528 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.567502 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/994ef2fb-3939-4282-a7b3-e08247dbf4f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "994ef2fb-3939-4282-a7b3-e08247dbf4f0" (UID: "994ef2fb-3939-4282-a7b3-e08247dbf4f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:23:18.567717 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.567700 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/994ef2fb-3939-4282-a7b3-e08247dbf4f0-model-cache\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:18.567782 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.567719 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/994ef2fb-3939-4282-a7b3-e08247dbf4f0-home\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:18.567782 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.567732 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/994ef2fb-3939-4282-a7b3-e08247dbf4f0-tmp-dir\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:18.569215 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.569193 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/994ef2fb-3939-4282-a7b3-e08247dbf4f0-dshm" (OuterVolumeSpecName: "dshm") pod "994ef2fb-3939-4282-a7b3-e08247dbf4f0" (UID: "994ef2fb-3939-4282-a7b3-e08247dbf4f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:23:18.569287 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.569245 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994ef2fb-3939-4282-a7b3-e08247dbf4f0-kube-api-access-rs7hj" (OuterVolumeSpecName: "kube-api-access-rs7hj") pod "994ef2fb-3939-4282-a7b3-e08247dbf4f0" (UID: "994ef2fb-3939-4282-a7b3-e08247dbf4f0"). InnerVolumeSpecName "kube-api-access-rs7hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:23:18.569287 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.569271 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/994ef2fb-3939-4282-a7b3-e08247dbf4f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "994ef2fb-3939-4282-a7b3-e08247dbf4f0" (UID: "994ef2fb-3939-4282-a7b3-e08247dbf4f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:23:18.668302 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.668269 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs7hj\" (UniqueName: \"kubernetes.io/projected/994ef2fb-3939-4282-a7b3-e08247dbf4f0-kube-api-access-rs7hj\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:18.668302 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.668297 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/994ef2fb-3939-4282-a7b3-e08247dbf4f0-dshm\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:18.668302 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:18.668307 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/994ef2fb-3939-4282-a7b3-e08247dbf4f0-tls-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:19.063645 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:19.063599 2556 patch_prober.go:28] interesting pod/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 14 11:23:19.063845 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:19.063665 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" podUID="51fd3d77-7af2-4683-8b6b-f6783b654cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 14 11:23:19.480873 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:19.480838 2556 generic.go:356] "Generic (PLEG): container finished" podID="986bc775-2278-492e-a152-25b8f4f12a5c" containerID="732e083df2d1cb08cef3349fad1739accf68bad8b48f60a26e5185aa725c8dd9" exitCode=0 Sep 14 11:23:19.481232 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:19.480922 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6d272c-krztg" event={"ID":"986bc775-2278-492e-a152-25b8f4f12a5c","Type":"ContainerDied","Data":"732e083df2d1cb08cef3349fad1739accf68bad8b48f60a26e5185aa725c8dd9"} Sep 14 11:23:19.481232 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:19.480954 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp" Sep 14 11:23:19.566158 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:19.566130 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp"] Sep 14 11:23:19.578281 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:19.578225 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6d272c/test-model-6d272c-kserve-5589c9fc5c-r4vdp"] Sep 14 11:23:19.712113 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:19.704705 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972"] Sep 14 11:23:20.063181 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:20.063142 2556 patch_prober.go:28] interesting pod/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 14 11:23:20.063343 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:20.063209 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" podUID="51fd3d77-7af2-4683-8b6b-f6783b654cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 14 11:23:20.384004 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:20.383960 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="994ef2fb-3939-4282-a7b3-e08247dbf4f0" path="/var/lib/kubelet/pods/994ef2fb-3939-4282-a7b3-e08247dbf4f0/volumes" Sep 14 11:23:20.619193 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:20.619171 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6d272c-krztg" Sep 14 11:23:20.714749 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:20.714715 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/986bc775-2278-492e-a152-25b8f4f12a5c-maas-api-service-ca\") pod \"986bc775-2278-492e-a152-25b8f4f12a5c\" (UID: \"986bc775-2278-492e-a152-25b8f4f12a5c\") " Sep 14 11:23:20.715048 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:20.715027 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/986bc775-2278-492e-a152-25b8f4f12a5c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "986bc775-2278-492e-a152-25b8f4f12a5c" (UID: "986bc775-2278-492e-a152-25b8f4f12a5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:23:20.815686 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:20.815659 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/986bc775-2278-492e-a152-25b8f4f12a5c-maas-api-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:21.066941 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:21.066909 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:23:21.068042 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:21.068026 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:23:21.496515 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:21.496486 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6d272c-krztg" Sep 14 11:23:21.496698 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:21.496529 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6d272c-krztg" event={"ID":"986bc775-2278-492e-a152-25b8f4f12a5c","Type":"ContainerDied","Data":"ea0abacbdbe6596f0ade01f27905405ed68782a0737c570a271b9cf336ea55cf"} Sep 14 11:23:21.496698 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:21.496582 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea0abacbdbe6596f0ade01f27905405ed68782a0737c570a271b9cf336ea55cf" Sep 14 11:23:21.498280 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:21.498253 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7935-58a94d" event={"ID":"e5ab06f8-f3d7-40aa-b02e-14b193d0d5b3","Type":"ContainerStarted","Data":"c2c17a86ca484751a80e8f39e8f9b7ccbc322d65e040bbfd66697f22eb0974b1"} Sep 14 11:23:21.535661 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:21.535621 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7935-58a94d" podStartSLOduration=2.696603223 podStartE2EDuration="5.535608421s" totalImagesPullingTime="2.839005198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:23:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:23:21.53162287 +0000 UTC m=+1359.595025780" watchObservedRunningTime="2026-09-14 11:23:21.535608421 +0000 UTC m=+1359.599011388" Sep 14 11:23:22.172507 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:22.172477 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7935-58a94d"] Sep 14 11:23:22.178366 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:22.178342 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7935-58a94d"] Sep 14 11:23:22.301305 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:22.301239 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7935-56a1bd"] Sep 14 11:23:22.301563 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:22.301552 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="986bc775-2278-492e-a152-25b8f4f12a5c" containerName="revoke-keys" Sep 14 11:23:22.301606 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:22.301565 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="986bc775-2278-492e-a152-25b8f4f12a5c" containerName="revoke-keys" Sep 14 11:23:22.301606 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:22.301577 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="994ef2fb-3939-4282-a7b3-e08247dbf4f0" containerName="main" Sep 14 11:23:22.301606 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:22.301583 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="994ef2fb-3939-4282-a7b3-e08247dbf4f0" containerName="main" Sep 14 11:23:22.301606 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:22.301599 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5ab06f8-f3d7-40aa-b02e-14b193d0d5b3" containerName="test-curl-7935-58a94d" Sep 14 11:23:22.301606 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:22.301604 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5ab06f8-f3d7-40aa-b02e-14b193d0d5b3" containerName="test-curl-7935-58a94d" Sep 14 11:23:22.301754 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:22.301653 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="994ef2fb-3939-4282-a7b3-e08247dbf4f0" containerName="main" Sep 14 11:23:22.301754 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:22.301662 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="986bc775-2278-492e-a152-25b8f4f12a5c" containerName="revoke-keys" Sep 14 11:23:22.301754 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:22.301671 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5ab06f8-f3d7-40aa-b02e-14b193d0d5b3" containerName="test-curl-7935-58a94d" Sep 14 11:23:22.303535 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:22.303517 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7935-56a1bd" Sep 14 11:23:22.321627 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:22.321606 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7935-56a1bd"] Sep 14 11:23:22.429417 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:22.429384 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5ljq\" (UniqueName: \"kubernetes.io/projected/8ff76866-baee-4066-9cf8-88257601eabb-kube-api-access-z5ljq\") pod \"test-curl-7935-56a1bd\" (UID: \"8ff76866-baee-4066-9cf8-88257601eabb\") " pod="opendatahub/test-curl-7935-56a1bd" Sep 14 11:23:22.503605 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:23:22.503581 2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7935-58a94d" container="test-curl-7935-58a94d" Sep 14 11:23:22.529841 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:22.529815 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5ljq\" (UniqueName: \"kubernetes.io/projected/8ff76866-baee-4066-9cf8-88257601eabb-kube-api-access-z5ljq\") pod \"test-curl-7935-56a1bd\" (UID: \"8ff76866-baee-4066-9cf8-88257601eabb\") " pod="opendatahub/test-curl-7935-56a1bd" Sep 14 11:23:22.541468 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:22.541439 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5ljq\" (UniqueName: \"kubernetes.io/projected/8ff76866-baee-4066-9cf8-88257601eabb-kube-api-access-z5ljq\") pod \"test-curl-7935-56a1bd\" (UID: \"8ff76866-baee-4066-9cf8-88257601eabb\") " pod="opendatahub/test-curl-7935-56a1bd" Sep 14 11:23:22.612493 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:22.612475 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7935-56a1bd" Sep 14 11:23:22.887142 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:22.887120 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7935-56a1bd"] Sep 14 11:23:22.888758 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:23:22.888730 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ff76866_baee_4066_9cf8_88257601eabb.slice/crio-dbdbce0d327e942978c34278c6e993ac36de52a172c0a2817620c6ca35091bb5 WatchSource:0}: Error finding container dbdbce0d327e942978c34278c6e993ac36de52a172c0a2817620c6ca35091bb5: Status 404 returned error can't find the container with id dbdbce0d327e942978c34278c6e993ac36de52a172c0a2817620c6ca35091bb5 Sep 14 11:23:23.509838 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:23.509798 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7935-56a1bd" event={"ID":"8ff76866-baee-4066-9cf8-88257601eabb","Type":"ContainerStarted","Data":"d235856e39f62e8d8a9c39678276bfff8eb69aef0f945bc12777d2f50c5c237e"} Sep 14 11:23:23.509838 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:23.509844 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7935-56a1bd" event={"ID":"8ff76866-baee-4066-9cf8-88257601eabb","Type":"ContainerStarted","Data":"dbdbce0d327e942978c34278c6e993ac36de52a172c0a2817620c6ca35091bb5"} Sep 14 11:23:23.510308 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:23.509894 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7935-58a94d" podUID="e5ab06f8-f3d7-40aa-b02e-14b193d0d5b3" containerName="test-curl-7935-58a94d" containerID="cri-o://c2c17a86ca484751a80e8f39e8f9b7ccbc322d65e040bbfd66697f22eb0974b1" gracePeriod=2 Sep 14 11:23:23.527931 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:23.527890 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7935-56a1bd" podStartSLOduration=1.527876442 podStartE2EDuration="1.527876442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:23:23.526549114 +0000 UTC m=+1361.589952024" watchObservedRunningTime="2026-09-14 11:23:23.527876442 +0000 UTC m=+1361.591279396" Sep 14 11:23:24.208549 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:24.208512 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7935-56a1bd"] Sep 14 11:23:24.211619 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:24.211559 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7935-56a1bd"] Sep 14 11:23:24.349164 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:24.349138 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7935-901c40"] Sep 14 11:23:24.349455 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:24.349444 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ff76866-baee-4066-9cf8-88257601eabb" containerName="test-curl-7935-56a1bd" Sep 14 11:23:24.349498 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:24.349456 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ff76866-baee-4066-9cf8-88257601eabb" containerName="test-curl-7935-56a1bd" Sep 14 11:23:24.349533 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:24.349517 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ff76866-baee-4066-9cf8-88257601eabb" containerName="test-curl-7935-56a1bd" Sep 14 11:23:24.357663 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:24.357640 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7935-901c40" Sep 14 11:23:24.361499 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:24.361471 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7935-901c40"] Sep 14 11:23:24.449491 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:24.449467 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shv6l\" (UniqueName: \"kubernetes.io/projected/0c14a4a6-8d2c-4768-97dc-f6bebccc0e17-kube-api-access-shv6l\") pod \"test-curl-7935-901c40\" (UID: \"0c14a4a6-8d2c-4768-97dc-f6bebccc0e17\") " pod="opendatahub/test-curl-7935-901c40" Sep 14 11:23:24.514876 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:23:24.514815 2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7935-56a1bd" container="test-curl-7935-56a1bd" Sep 14 11:23:24.550488 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:24.550458 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shv6l\" (UniqueName: \"kubernetes.io/projected/0c14a4a6-8d2c-4768-97dc-f6bebccc0e17-kube-api-access-shv6l\") pod \"test-curl-7935-901c40\" (UID: \"0c14a4a6-8d2c-4768-97dc-f6bebccc0e17\") " pod="opendatahub/test-curl-7935-901c40" Sep 14 11:23:24.560554 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:24.560530 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shv6l\" (UniqueName: \"kubernetes.io/projected/0c14a4a6-8d2c-4768-97dc-f6bebccc0e17-kube-api-access-shv6l\") pod \"test-curl-7935-901c40\" (UID: \"0c14a4a6-8d2c-4768-97dc-f6bebccc0e17\") " pod="opendatahub/test-curl-7935-901c40" Sep 14 11:23:24.666913 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:24.666891 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7935-901c40" Sep 14 11:23:24.962399 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:24.962371 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7935-901c40"] Sep 14 11:23:24.966460 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:23:24.966438 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c14a4a6_8d2c_4768_97dc_f6bebccc0e17.slice/crio-cbc653e800ef95f2f9721bd9fdced326cbedd685e89f1cf0572e5ada2ad977a0 WatchSource:0}: Error finding container cbc653e800ef95f2f9721bd9fdced326cbedd685e89f1cf0572e5ada2ad977a0: Status 404 returned error can't find the container with id cbc653e800ef95f2f9721bd9fdced326cbedd685e89f1cf0572e5ada2ad977a0 Sep 14 11:23:25.522044 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:25.522017 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7935-901c40" event={"ID":"0c14a4a6-8d2c-4768-97dc-f6bebccc0e17","Type":"ContainerStarted","Data":"8ffbe015c1d06b2308c08e671bd9d1f6b78a53767550223083d172e9659c1333"} Sep 14 11:23:25.522415 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:25.522053 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7935-901c40" event={"ID":"0c14a4a6-8d2c-4768-97dc-f6bebccc0e17","Type":"ContainerStarted","Data":"cbc653e800ef95f2f9721bd9fdced326cbedd685e89f1cf0572e5ada2ad977a0"} Sep 14 11:23:25.522415 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:25.522174 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7935-56a1bd" podUID="8ff76866-baee-4066-9cf8-88257601eabb" containerName="test-curl-7935-56a1bd" containerID="cri-o://d235856e39f62e8d8a9c39678276bfff8eb69aef0f945bc12777d2f50c5c237e" gracePeriod=2 Sep 14 11:23:25.542810 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:25.542767 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7935-901c40" podStartSLOduration=1.542752457 podStartE2EDuration="1.542752457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:23:25.542689919 +0000 UTC m=+1363.606092830" watchObservedRunningTime="2026-09-14 11:23:25.542752457 +0000 UTC m=+1363.606155366" Sep 14 11:23:25.692991 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:25.692959 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7935-58a94d" Sep 14 11:23:25.861241 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:25.861218 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmphv\" (UniqueName: \"kubernetes.io/projected/e5ab06f8-f3d7-40aa-b02e-14b193d0d5b3-kube-api-access-fmphv\") pod \"e5ab06f8-f3d7-40aa-b02e-14b193d0d5b3\" (UID: \"e5ab06f8-f3d7-40aa-b02e-14b193d0d5b3\") " Sep 14 11:23:25.863207 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:25.863183 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ab06f8-f3d7-40aa-b02e-14b193d0d5b3-kube-api-access-fmphv" (OuterVolumeSpecName: "kube-api-access-fmphv") pod "e5ab06f8-f3d7-40aa-b02e-14b193d0d5b3" (UID: "e5ab06f8-f3d7-40aa-b02e-14b193d0d5b3"). InnerVolumeSpecName "kube-api-access-fmphv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:23:25.962419 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:25.962396 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmphv\" (UniqueName: \"kubernetes.io/projected/e5ab06f8-f3d7-40aa-b02e-14b193d0d5b3-kube-api-access-fmphv\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:26.219018 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:26.218943 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7935-901c40"] Sep 14 11:23:26.222730 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:26.222707 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7935-901c40"] Sep 14 11:23:26.356893 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:26.356867 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7935-364366"] Sep 14 11:23:26.357211 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:26.357197 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c14a4a6-8d2c-4768-97dc-f6bebccc0e17" containerName="test-curl-7935-901c40" Sep 14 11:23:26.357271 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:26.357214 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c14a4a6-8d2c-4768-97dc-f6bebccc0e17" containerName="test-curl-7935-901c40" Sep 14 11:23:26.357271 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:26.357269 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c14a4a6-8d2c-4768-97dc-f6bebccc0e17" containerName="test-curl-7935-901c40" Sep 14 11:23:26.359314 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:26.359298 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7935-364366" Sep 14 11:23:26.369168 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:26.369146 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7935-364366"] Sep 14 11:23:26.384369 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:26.384339 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5ab06f8-f3d7-40aa-b02e-14b193d0d5b3" path="/var/lib/kubelet/pods/e5ab06f8-f3d7-40aa-b02e-14b193d0d5b3/volumes" Sep 14 11:23:26.466367 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:26.466336 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk6rd\" (UniqueName: \"kubernetes.io/projected/89e0e35b-a29c-46bf-90c4-08fdf038269c-kube-api-access-kk6rd\") pod \"test-curl-7935-364366\" (UID: \"89e0e35b-a29c-46bf-90c4-08fdf038269c\") " pod="opendatahub/test-curl-7935-364366" Sep 14 11:23:26.528770 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:26.528710 2556 generic.go:356] "Generic (PLEG): container finished" podID="e5ab06f8-f3d7-40aa-b02e-14b193d0d5b3" containerID="c2c17a86ca484751a80e8f39e8f9b7ccbc322d65e040bbfd66697f22eb0974b1" exitCode=137 Sep 14 11:23:26.528770 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:26.528757 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7935-58a94d" Sep 14 11:23:26.529163 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:26.528812 2556 scope.go:117] "RemoveContainer" containerID="c2c17a86ca484751a80e8f39e8f9b7ccbc322d65e040bbfd66697f22eb0974b1" Sep 14 11:23:26.529163 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:23:26.529045 2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7935-901c40" container="test-curl-7935-901c40" Sep 14 11:23:26.537144 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:26.537128 2556 scope.go:117] "RemoveContainer" containerID="c2c17a86ca484751a80e8f39e8f9b7ccbc322d65e040bbfd66697f22eb0974b1" Sep 14 11:23:26.537405 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:23:26.537389 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c17a86ca484751a80e8f39e8f9b7ccbc322d65e040bbfd66697f22eb0974b1\": container with ID starting with c2c17a86ca484751a80e8f39e8f9b7ccbc322d65e040bbfd66697f22eb0974b1 not found: ID does not exist" containerID="c2c17a86ca484751a80e8f39e8f9b7ccbc322d65e040bbfd66697f22eb0974b1" Sep 14 11:23:26.537452 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:26.537414 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c17a86ca484751a80e8f39e8f9b7ccbc322d65e040bbfd66697f22eb0974b1"} err="failed to get container status \"c2c17a86ca484751a80e8f39e8f9b7ccbc322d65e040bbfd66697f22eb0974b1\": rpc error: code = NotFound desc = could not find container \"c2c17a86ca484751a80e8f39e8f9b7ccbc322d65e040bbfd66697f22eb0974b1\": container with ID starting with c2c17a86ca484751a80e8f39e8f9b7ccbc322d65e040bbfd66697f22eb0974b1 not found: ID does not exist" Sep 14 11:23:26.566957 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:26.566936 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk6rd\" (UniqueName: \"kubernetes.io/projected/89e0e35b-a29c-46bf-90c4-08fdf038269c-kube-api-access-kk6rd\") pod \"test-curl-7935-364366\" (UID: \"89e0e35b-a29c-46bf-90c4-08fdf038269c\") " pod="opendatahub/test-curl-7935-364366" Sep 14 11:23:26.581278 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:26.581256 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk6rd\" (UniqueName: \"kubernetes.io/projected/89e0e35b-a29c-46bf-90c4-08fdf038269c-kube-api-access-kk6rd\") pod \"test-curl-7935-364366\" (UID: \"89e0e35b-a29c-46bf-90c4-08fdf038269c\") " pod="opendatahub/test-curl-7935-364366" Sep 14 11:23:26.668330 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:26.668309 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7935-364366" Sep 14 11:23:26.945154 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:26.945130 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7935-364366"] Sep 14 11:23:26.947515 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:23:26.947488 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89e0e35b_a29c_46bf_90c4_08fdf038269c.slice/crio-0cebeab4a828c8f574d256e10a3b2dc566efae4d78730054ed4d292abbefeb2b WatchSource:0}: Error finding container 0cebeab4a828c8f574d256e10a3b2dc566efae4d78730054ed4d292abbefeb2b: Status 404 returned error can't find the container with id 0cebeab4a828c8f574d256e10a3b2dc566efae4d78730054ed4d292abbefeb2b Sep 14 11:23:27.535791 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:27.535766 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7935-364366" event={"ID":"89e0e35b-a29c-46bf-90c4-08fdf038269c","Type":"ContainerStarted","Data":"d22a20dd23ad02210ad37235227bb8db20d8b3788ab5a650d3baa10b472c765e"} Sep 14 11:23:27.536076 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:27.535801 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7935-364366" event={"ID":"89e0e35b-a29c-46bf-90c4-08fdf038269c","Type":"ContainerStarted","Data":"0cebeab4a828c8f574d256e10a3b2dc566efae4d78730054ed4d292abbefeb2b"} Sep 14 11:23:27.536076 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:27.535925 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7935-901c40" podUID="0c14a4a6-8d2c-4768-97dc-f6bebccc0e17" containerName="test-curl-7935-901c40" containerID="cri-o://8ffbe015c1d06b2308c08e671bd9d1f6b78a53767550223083d172e9659c1333" gracePeriod=2 Sep 14 11:23:27.553770 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:27.553729 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7935-364366" podStartSLOduration=1.553715867 podStartE2EDuration="1.553715867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:23:27.5519343 +0000 UTC m=+1365.615337209" watchObservedRunningTime="2026-09-14 11:23:27.553715867 +0000 UTC m=+1365.617118776" Sep 14 11:23:27.709557 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:27.709535 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7935-56a1bd" Sep 14 11:23:27.878494 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:27.878466 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5ljq\" (UniqueName: \"kubernetes.io/projected/8ff76866-baee-4066-9cf8-88257601eabb-kube-api-access-z5ljq\") pod \"8ff76866-baee-4066-9cf8-88257601eabb\" (UID: \"8ff76866-baee-4066-9cf8-88257601eabb\") " Sep 14 11:23:27.884891 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:27.880702 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff76866-baee-4066-9cf8-88257601eabb-kube-api-access-z5ljq" (OuterVolumeSpecName: "kube-api-access-z5ljq") pod "8ff76866-baee-4066-9cf8-88257601eabb" (UID: "8ff76866-baee-4066-9cf8-88257601eabb"). InnerVolumeSpecName "kube-api-access-z5ljq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:23:27.979851 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:27.979827 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5ljq\" (UniqueName: \"kubernetes.io/projected/8ff76866-baee-4066-9cf8-88257601eabb-kube-api-access-z5ljq\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:28.206124 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:28.206035 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7935-364366"] Sep 14 11:23:28.212246 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:28.212216 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7935-364366"] Sep 14 11:23:28.319249 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:28.319214 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6d272c-krztg"] Sep 14 11:23:28.322863 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:28.322837 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6d272c-krztg"] Sep 14 11:23:28.383446 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:28.383419 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ff76866-baee-4066-9cf8-88257601eabb" path="/var/lib/kubelet/pods/8ff76866-baee-4066-9cf8-88257601eabb/volumes" Sep 14 11:23:28.383758 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:28.383744 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="986bc775-2278-492e-a152-25b8f4f12a5c" path="/var/lib/kubelet/pods/986bc775-2278-492e-a152-25b8f4f12a5c/volumes" Sep 14 11:23:28.542122 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:28.542028 2556 generic.go:356] "Generic (PLEG): container finished" podID="8ff76866-baee-4066-9cf8-88257601eabb" containerID="d235856e39f62e8d8a9c39678276bfff8eb69aef0f945bc12777d2f50c5c237e" exitCode=137 Sep 14 11:23:28.542122 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:28.542083 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7935-56a1bd" Sep 14 11:23:28.542553 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:28.542132 2556 scope.go:117] "RemoveContainer" containerID="d235856e39f62e8d8a9c39678276bfff8eb69aef0f945bc12777d2f50c5c237e" Sep 14 11:23:28.542553 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:23:28.542318 2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7935-364366" container="test-curl-7935-364366" Sep 14 11:23:28.550245 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:28.550228 2556 scope.go:117] "RemoveContainer" containerID="d235856e39f62e8d8a9c39678276bfff8eb69aef0f945bc12777d2f50c5c237e" Sep 14 11:23:28.550508 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:23:28.550489 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d235856e39f62e8d8a9c39678276bfff8eb69aef0f945bc12777d2f50c5c237e\": container with ID starting with d235856e39f62e8d8a9c39678276bfff8eb69aef0f945bc12777d2f50c5c237e not found: ID does not exist" containerID="d235856e39f62e8d8a9c39678276bfff8eb69aef0f945bc12777d2f50c5c237e" Sep 14 11:23:28.550572 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:28.550514 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d235856e39f62e8d8a9c39678276bfff8eb69aef0f945bc12777d2f50c5c237e"} err="failed to get container status \"d235856e39f62e8d8a9c39678276bfff8eb69aef0f945bc12777d2f50c5c237e\": rpc error: code = NotFound desc = could not find container \"d235856e39f62e8d8a9c39678276bfff8eb69aef0f945bc12777d2f50c5c237e\": container with ID starting with d235856e39f62e8d8a9c39678276bfff8eb69aef0f945bc12777d2f50c5c237e not found: ID does not exist" Sep 14 11:23:29.254111 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.251141 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr"] Sep 14 11:23:29.254111 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.251698 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89e0e35b-a29c-46bf-90c4-08fdf038269c" containerName="test-curl-7935-364366" Sep 14 11:23:29.254111 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.251717 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="89e0e35b-a29c-46bf-90c4-08fdf038269c" containerName="test-curl-7935-364366" Sep 14 11:23:29.254111 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.251807 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="89e0e35b-a29c-46bf-90c4-08fdf038269c" containerName="test-curl-7935-364366" Sep 14 11:23:29.266848 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.263328 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.269900 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.269876 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7cc544b6-openshift-default-dockercfg-zt955\"" Sep 14 11:23:29.280938 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.280915 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr"] Sep 14 11:23:29.393482 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.393442 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istio-token\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.393731 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.393696 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-workload-certs\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.393881 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.393839 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-credential-socket\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.394029 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.393958 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istio-podinfo\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.394029 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.394004 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istiod-ca-cert\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.394137 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.394074 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt8n9\" (UniqueName: \"kubernetes.io/projected/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-kube-api-access-qt8n9\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.394202 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.394134 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istio-data\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.394202 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.394175 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istio-envoy\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.394300 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.394235 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-workload-socket\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.495511 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.495478 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-credential-socket\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.495671 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.495526 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istio-podinfo\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.495671 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.495558 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istiod-ca-cert\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.495671 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.495582 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt8n9\" (UniqueName: \"kubernetes.io/projected/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-kube-api-access-qt8n9\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.495671 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.495600 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istio-data\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.495671 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.495620 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istio-envoy\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.495671 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.495653 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-workload-socket\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.495959 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.495691 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istio-token\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.495959 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.495727 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-workload-certs\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.496057 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.495952 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-credential-socket\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.496123 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.496063 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istio-data\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.496167 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.496116 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-workload-socket\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.496356 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.496336 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-workload-certs\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.496513 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.496497 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istiod-ca-cert\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.497989 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.497963 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istio-envoy\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.498113 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.498041 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istio-podinfo\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.523112 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.523029 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istio-token\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.523112 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.523029 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt8n9\" (UniqueName: \"kubernetes.io/projected/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-kube-api-access-qt8n9\") pod \"e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.549324 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.549268 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7935-364366" podUID="89e0e35b-a29c-46bf-90c4-08fdf038269c" containerName="test-curl-7935-364366" containerID="cri-o://d22a20dd23ad02210ad37235227bb8db20d8b3788ab5a650d3baa10b472c765e" gracePeriod=2 Sep 14 11:23:29.575294 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.575273 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:29.726332 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.726308 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7935-901c40" Sep 14 11:23:29.859009 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.858967 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr"] Sep 14 11:23:29.862062 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:23:29.862036 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1b2d34_18c4_4f8c_80fe_55bd8d5ecef6.slice/crio-00b57b98b770dd32fa6f5b89cd99747955176150d7a386acc4e44c19bf80d1e9 WatchSource:0}: Error finding container 00b57b98b770dd32fa6f5b89cd99747955176150d7a386acc4e44c19bf80d1e9: Status 404 returned error can't find the container with id 00b57b98b770dd32fa6f5b89cd99747955176150d7a386acc4e44c19bf80d1e9 Sep 14 11:23:29.864903 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.864867 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:23:29.865008 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.864937 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:23:29.865008 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.864967 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:23:29.900022 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.900000 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shv6l\" (UniqueName: \"kubernetes.io/projected/0c14a4a6-8d2c-4768-97dc-f6bebccc0e17-kube-api-access-shv6l\") pod \"0c14a4a6-8d2c-4768-97dc-f6bebccc0e17\" (UID: \"0c14a4a6-8d2c-4768-97dc-f6bebccc0e17\") " Sep 14 11:23:29.901875 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:29.901852 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c14a4a6-8d2c-4768-97dc-f6bebccc0e17-kube-api-access-shv6l" (OuterVolumeSpecName: "kube-api-access-shv6l") pod "0c14a4a6-8d2c-4768-97dc-f6bebccc0e17" (UID: "0c14a4a6-8d2c-4768-97dc-f6bebccc0e17"). InnerVolumeSpecName "kube-api-access-shv6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:23:30.001236 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:30.001207 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shv6l\" (UniqueName: \"kubernetes.io/projected/0c14a4a6-8d2c-4768-97dc-f6bebccc0e17-kube-api-access-shv6l\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:30.385368 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:30.385333 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c14a4a6-8d2c-4768-97dc-f6bebccc0e17" path="/var/lib/kubelet/pods/0c14a4a6-8d2c-4768-97dc-f6bebccc0e17/volumes" Sep 14 11:23:30.556029 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:30.555995 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" event={"ID":"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6","Type":"ContainerStarted","Data":"de5c3cf8e282799d90e34a35bcfb42e32c9a68ecd903bf8af3ad3935c6b09f2c"} Sep 14 11:23:30.556546 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:30.556055 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" event={"ID":"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6","Type":"ContainerStarted","Data":"00b57b98b770dd32fa6f5b89cd99747955176150d7a386acc4e44c19bf80d1e9"} Sep 14 11:23:30.557491 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:30.557465 2556 generic.go:356] "Generic (PLEG): container finished" podID="0c14a4a6-8d2c-4768-97dc-f6bebccc0e17" containerID="8ffbe015c1d06b2308c08e671bd9d1f6b78a53767550223083d172e9659c1333" exitCode=137 Sep 14 11:23:30.557620 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:30.557531 2556 scope.go:117] "RemoveContainer" containerID="8ffbe015c1d06b2308c08e671bd9d1f6b78a53767550223083d172e9659c1333" Sep 14 11:23:30.557620 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:30.557527 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7935-901c40" Sep 14 11:23:30.566022 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:30.565996 2556 scope.go:117] "RemoveContainer" containerID="8ffbe015c1d06b2308c08e671bd9d1f6b78a53767550223083d172e9659c1333" Sep 14 11:23:30.566327 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:23:30.566308 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ffbe015c1d06b2308c08e671bd9d1f6b78a53767550223083d172e9659c1333\": container with ID starting with 8ffbe015c1d06b2308c08e671bd9d1f6b78a53767550223083d172e9659c1333 not found: ID does not exist" containerID="8ffbe015c1d06b2308c08e671bd9d1f6b78a53767550223083d172e9659c1333" Sep 14 11:23:30.566389 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:30.566337 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ffbe015c1d06b2308c08e671bd9d1f6b78a53767550223083d172e9659c1333"} err="failed to get container status \"8ffbe015c1d06b2308c08e671bd9d1f6b78a53767550223083d172e9659c1333\": rpc error: code = NotFound desc = could not find container \"8ffbe015c1d06b2308c08e671bd9d1f6b78a53767550223083d172e9659c1333\": container with ID starting with 8ffbe015c1d06b2308c08e671bd9d1f6b78a53767550223083d172e9659c1333 not found: ID does not exist" Sep 14 11:23:30.575442 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:30.575420 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:30.580435 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:30.580418 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:30.583518 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:30.583473 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" podStartSLOduration=1.58345947 podStartE2EDuration="1.58345947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:23:30.581404783 +0000 UTC m=+1368.644807695" watchObservedRunningTime="2026-09-14 11:23:30.58345947 +0000 UTC m=+1368.646862380" Sep 14 11:23:31.564702 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:31.564684 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:31.565541 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:31.565527 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:31.734489 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:31.734467 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7935-364366" Sep 14 11:23:31.922777 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:31.922752 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kk6rd\" (UniqueName: \"kubernetes.io/projected/89e0e35b-a29c-46bf-90c4-08fdf038269c-kube-api-access-kk6rd\") pod \"89e0e35b-a29c-46bf-90c4-08fdf038269c\" (UID: \"89e0e35b-a29c-46bf-90c4-08fdf038269c\") " Sep 14 11:23:31.924668 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:31.924635 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e0e35b-a29c-46bf-90c4-08fdf038269c-kube-api-access-kk6rd" (OuterVolumeSpecName: "kube-api-access-kk6rd") pod "89e0e35b-a29c-46bf-90c4-08fdf038269c" (UID: "89e0e35b-a29c-46bf-90c4-08fdf038269c"). InnerVolumeSpecName "kube-api-access-kk6rd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:23:32.023608 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:32.023579 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk6rd\" (UniqueName: \"kubernetes.io/projected/89e0e35b-a29c-46bf-90c4-08fdf038269c-kube-api-access-kk6rd\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:32.384476 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:32.384435 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89e0e35b-a29c-46bf-90c4-08fdf038269c" path="/var/lib/kubelet/pods/89e0e35b-a29c-46bf-90c4-08fdf038269c/volumes" Sep 14 11:23:32.570435 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:32.570404 2556 generic.go:356] "Generic (PLEG): container finished" podID="89e0e35b-a29c-46bf-90c4-08fdf038269c" containerID="d22a20dd23ad02210ad37235227bb8db20d8b3788ab5a650d3baa10b472c765e" exitCode=137 Sep 14 11:23:32.570798 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:32.570457 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7935-364366" Sep 14 11:23:32.570798 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:32.570496 2556 scope.go:117] "RemoveContainer" containerID="d22a20dd23ad02210ad37235227bb8db20d8b3788ab5a650d3baa10b472c765e" Sep 14 11:23:32.578719 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:32.578701 2556 scope.go:117] "RemoveContainer" containerID="d22a20dd23ad02210ad37235227bb8db20d8b3788ab5a650d3baa10b472c765e" Sep 14 11:23:32.578961 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:23:32.578947 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d22a20dd23ad02210ad37235227bb8db20d8b3788ab5a650d3baa10b472c765e\": container with ID starting with d22a20dd23ad02210ad37235227bb8db20d8b3788ab5a650d3baa10b472c765e not found: ID does not exist" containerID="d22a20dd23ad02210ad37235227bb8db20d8b3788ab5a650d3baa10b472c765e" Sep 14 11:23:32.579030 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:32.578966 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d22a20dd23ad02210ad37235227bb8db20d8b3788ab5a650d3baa10b472c765e"} err="failed to get container status \"d22a20dd23ad02210ad37235227bb8db20d8b3788ab5a650d3baa10b472c765e\": rpc error: code = NotFound desc = could not find container \"d22a20dd23ad02210ad37235227bb8db20d8b3788ab5a650d3baa10b472c765e\": container with ID starting with d22a20dd23ad02210ad37235227bb8db20d8b3788ab5a650d3baa10b472c765e not found: ID does not exist" Sep 14 11:23:34.729739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:34.729692 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" podUID="f6a21e8f-68d0-47ba-b030-81f39e58b460" containerName="main" containerID="cri-o://462c00bc7ee415f744d303a41929a32ea15a550d76c79d2c6aed6bd5aad587ee" gracePeriod=45 Sep 14 11:23:34.980910 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:34.980853 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:23:35.150844 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.150809 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6cxd\" (UniqueName: \"kubernetes.io/projected/f6a21e8f-68d0-47ba-b030-81f39e58b460-kube-api-access-w6cxd\") pod \"f6a21e8f-68d0-47ba-b030-81f39e58b460\" (UID: \"f6a21e8f-68d0-47ba-b030-81f39e58b460\") " Sep 14 11:23:35.150985 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.150857 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a21e8f-68d0-47ba-b030-81f39e58b460-tls-certs\") pod \"f6a21e8f-68d0-47ba-b030-81f39e58b460\" (UID: \"f6a21e8f-68d0-47ba-b030-81f39e58b460\") " Sep 14 11:23:35.150985 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.150933 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a21e8f-68d0-47ba-b030-81f39e58b460-model-cache\") pod \"f6a21e8f-68d0-47ba-b030-81f39e58b460\" (UID: \"f6a21e8f-68d0-47ba-b030-81f39e58b460\") " Sep 14 11:23:35.150985 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.150972 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a21e8f-68d0-47ba-b030-81f39e58b460-home\") pod \"f6a21e8f-68d0-47ba-b030-81f39e58b460\" (UID: \"f6a21e8f-68d0-47ba-b030-81f39e58b460\") " Sep 14 11:23:35.151168 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.151021 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a21e8f-68d0-47ba-b030-81f39e58b460-dshm\") pod \"f6a21e8f-68d0-47ba-b030-81f39e58b460\" (UID: \"f6a21e8f-68d0-47ba-b030-81f39e58b460\") " Sep 14 11:23:35.151168 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.151049 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a21e8f-68d0-47ba-b030-81f39e58b460-tmp-dir\") pod \"f6a21e8f-68d0-47ba-b030-81f39e58b460\" (UID: \"f6a21e8f-68d0-47ba-b030-81f39e58b460\") " Sep 14 11:23:35.151322 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.151274 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a21e8f-68d0-47ba-b030-81f39e58b460-home" (OuterVolumeSpecName: "home") pod "f6a21e8f-68d0-47ba-b030-81f39e58b460" (UID: "f6a21e8f-68d0-47ba-b030-81f39e58b460"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:23:35.151322 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.151308 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a21e8f-68d0-47ba-b030-81f39e58b460-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6a21e8f-68d0-47ba-b030-81f39e58b460" (UID: "f6a21e8f-68d0-47ba-b030-81f39e58b460"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:23:35.151434 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.151352 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a21e8f-68d0-47ba-b030-81f39e58b460-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6a21e8f-68d0-47ba-b030-81f39e58b460" (UID: "f6a21e8f-68d0-47ba-b030-81f39e58b460"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:23:35.151474 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.151459 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a21e8f-68d0-47ba-b030-81f39e58b460-model-cache\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:35.151515 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.151480 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a21e8f-68d0-47ba-b030-81f39e58b460-home\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:35.151515 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.151493 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a21e8f-68d0-47ba-b030-81f39e58b460-tmp-dir\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:35.153036 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.153016 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a21e8f-68d0-47ba-b030-81f39e58b460-dshm" (OuterVolumeSpecName: "dshm") pod "f6a21e8f-68d0-47ba-b030-81f39e58b460" (UID: "f6a21e8f-68d0-47ba-b030-81f39e58b460"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:23:35.153135 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.153077 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a21e8f-68d0-47ba-b030-81f39e58b460-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6a21e8f-68d0-47ba-b030-81f39e58b460" (UID: "f6a21e8f-68d0-47ba-b030-81f39e58b460"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:23:35.153135 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.153114 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a21e8f-68d0-47ba-b030-81f39e58b460-kube-api-access-w6cxd" (OuterVolumeSpecName: "kube-api-access-w6cxd") pod "f6a21e8f-68d0-47ba-b030-81f39e58b460" (UID: "f6a21e8f-68d0-47ba-b030-81f39e58b460"). InnerVolumeSpecName "kube-api-access-w6cxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:23:35.252605 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.252548 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6cxd\" (UniqueName: \"kubernetes.io/projected/f6a21e8f-68d0-47ba-b030-81f39e58b460-kube-api-access-w6cxd\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:35.252605 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.252569 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a21e8f-68d0-47ba-b030-81f39e58b460-tls-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:35.252605 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.252579 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a21e8f-68d0-47ba-b030-81f39e58b460-dshm\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:35.592385 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.592298 2556 generic.go:356] "Generic (PLEG): container finished" podID="f6a21e8f-68d0-47ba-b030-81f39e58b460" containerID="462c00bc7ee415f744d303a41929a32ea15a550d76c79d2c6aed6bd5aad587ee" exitCode=0 Sep 14 11:23:35.592385 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.592353 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" Sep 14 11:23:35.592605 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.592386 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" event={"ID":"f6a21e8f-68d0-47ba-b030-81f39e58b460","Type":"ContainerDied","Data":"462c00bc7ee415f744d303a41929a32ea15a550d76c79d2c6aed6bd5aad587ee"} Sep 14 11:23:35.592605 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.592461 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972" event={"ID":"f6a21e8f-68d0-47ba-b030-81f39e58b460","Type":"ContainerDied","Data":"0f4d3ea7635797f929d7f3c631a92477e9b6951bbe6314dfdf371eb911e7f982"} Sep 14 11:23:35.592605 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.592485 2556 scope.go:117] "RemoveContainer" containerID="462c00bc7ee415f744d303a41929a32ea15a550d76c79d2c6aed6bd5aad587ee" Sep 14 11:23:35.601495 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.601475 2556 scope.go:117] "RemoveContainer" containerID="462c00bc7ee415f744d303a41929a32ea15a550d76c79d2c6aed6bd5aad587ee" Sep 14 11:23:35.601765 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:23:35.601747 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"462c00bc7ee415f744d303a41929a32ea15a550d76c79d2c6aed6bd5aad587ee\": container with ID starting with 462c00bc7ee415f744d303a41929a32ea15a550d76c79d2c6aed6bd5aad587ee not found: ID does not exist" containerID="462c00bc7ee415f744d303a41929a32ea15a550d76c79d2c6aed6bd5aad587ee" Sep 14 11:23:35.601805 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.601773 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"462c00bc7ee415f744d303a41929a32ea15a550d76c79d2c6aed6bd5aad587ee"} err="failed to get container status \"462c00bc7ee415f744d303a41929a32ea15a550d76c79d2c6aed6bd5aad587ee\": rpc error: code = NotFound desc = could not find container \"462c00bc7ee415f744d303a41929a32ea15a550d76c79d2c6aed6bd5aad587ee\": container with ID starting with 462c00bc7ee415f744d303a41929a32ea15a550d76c79d2c6aed6bd5aad587ee not found: ID does not exist" Sep 14 11:23:35.619214 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.619192 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972"] Sep 14 11:23:35.627441 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.627422 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-39ab34/test-model-39ab34-kserve-6b4c74959c-d7972"] Sep 14 11:23:35.858044 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.858016 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr"] Sep 14 11:23:35.858431 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:35.858311 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" podUID="9f43dd7d-23a3-4749-a795-562f8a28497b" containerName="istio-proxy" containerID="cri-o://c5d2f86ecca952cd91bd4209f41ce2ec7020fee9080eaae7d53c2ca8a04a66f1" gracePeriod=30 Sep 14 11:23:36.137442 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.137371 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj"] Sep 14 11:23:36.137713 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.137701 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6a21e8f-68d0-47ba-b030-81f39e58b460" containerName="main" Sep 14 11:23:36.137757 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.137715 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6a21e8f-68d0-47ba-b030-81f39e58b460" containerName="main" Sep 14 11:23:36.137791 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.137775 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6a21e8f-68d0-47ba-b030-81f39e58b460" containerName="main" Sep 14 11:23:36.147161 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.147135 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj" Sep 14 11:23:36.150719 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.150695 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7cc544b6\"" Sep 14 11:23:36.152668 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.152645 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj"] Sep 14 11:23:36.216064 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.216036 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39ab34-skcgr"] Sep 14 11:23:36.218277 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.218261 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39ab34-skcgr" Sep 14 11:23:36.221136 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.221116 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f98vt\"" Sep 14 11:23:36.221240 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.221141 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 11:23:36.228977 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.228956 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39ab34-skcgr"] Sep 14 11:23:36.284110 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.279311 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6327e7cc-7014-4b42-9207-1ff80b041bca-plugins-config-volume\") pod \"payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj\" (UID: \"6327e7cc-7014-4b42-9207-1ff80b041bca\") " pod="openshift-ingress/payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj" Sep 14 11:23:36.284110 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.279916 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9mjl\" (UniqueName: \"kubernetes.io/projected/6327e7cc-7014-4b42-9207-1ff80b041bca-kube-api-access-n9mjl\") pod \"payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj\" (UID: \"6327e7cc-7014-4b42-9207-1ff80b041bca\") " pod="openshift-ingress/payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj" Sep 14 11:23:36.380743 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.380697 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9mjl\" (UniqueName: \"kubernetes.io/projected/6327e7cc-7014-4b42-9207-1ff80b041bca-kube-api-access-n9mjl\") pod \"payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj\" (UID: \"6327e7cc-7014-4b42-9207-1ff80b041bca\") " pod="openshift-ingress/payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj" Sep 14 11:23:36.380743 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.380746 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4977506f-95a4-476c-80f6-8664a52d7f15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-39ab34-skcgr\" (UID: \"4977506f-95a4-476c-80f6-8664a52d7f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39ab34-skcgr" Sep 14 11:23:36.380982 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.380904 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6327e7cc-7014-4b42-9207-1ff80b041bca-plugins-config-volume\") pod \"payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj\" (UID: \"6327e7cc-7014-4b42-9207-1ff80b041bca\") " pod="openshift-ingress/payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj" Sep 14 11:23:36.381656 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.381637 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6327e7cc-7014-4b42-9207-1ff80b041bca-plugins-config-volume\") pod \"payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj\" (UID: \"6327e7cc-7014-4b42-9207-1ff80b041bca\") " pod="openshift-ingress/payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj" Sep 14 11:23:36.382805 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.382784 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6a21e8f-68d0-47ba-b030-81f39e58b460" path="/var/lib/kubelet/pods/f6a21e8f-68d0-47ba-b030-81f39e58b460/volumes" Sep 14 11:23:36.391036 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.390972 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9mjl\" (UniqueName: \"kubernetes.io/projected/6327e7cc-7014-4b42-9207-1ff80b041bca-kube-api-access-n9mjl\") pod \"payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj\" (UID: \"6327e7cc-7014-4b42-9207-1ff80b041bca\") " pod="openshift-ingress/payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj" Sep 14 11:23:36.459203 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.459182 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj" Sep 14 11:23:36.481668 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.481640 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4977506f-95a4-476c-80f6-8664a52d7f15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-39ab34-skcgr\" (UID: \"4977506f-95a4-476c-80f6-8664a52d7f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39ab34-skcgr" Sep 14 11:23:36.482235 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.482215 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4977506f-95a4-476c-80f6-8664a52d7f15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-39ab34-skcgr\" (UID: \"4977506f-95a4-476c-80f6-8664a52d7f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39ab34-skcgr" Sep 14 11:23:36.527348 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.527326 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39ab34-skcgr" Sep 14 11:23:36.733573 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.733549 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj"] Sep 14 11:23:36.735594 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:23:36.735570 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6327e7cc_7014_4b42_9207_1ff80b041bca.slice/crio-262cb49ec899d172b67016d6502e29bbf8843392533b8907b346fb7a40b3cc7f WatchSource:0}: Error finding container 262cb49ec899d172b67016d6502e29bbf8843392533b8907b346fb7a40b3cc7f: Status 404 returned error can't find the container with id 262cb49ec899d172b67016d6502e29bbf8843392533b8907b346fb7a40b3cc7f Sep 14 11:23:36.807425 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:36.807397 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39ab34-skcgr"] Sep 14 11:23:36.809052 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:23:36.809025 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4977506f_95a4_476c_80f6_8664a52d7f15.slice/crio-469be408c0730343b7c8200be252d21d50beb51feddfe2e607d2ceae5bed05e6 WatchSource:0}: Error finding container 469be408c0730343b7c8200be252d21d50beb51feddfe2e607d2ceae5bed05e6: Status 404 returned error can't find the container with id 469be408c0730343b7c8200be252d21d50beb51feddfe2e607d2ceae5bed05e6 Sep 14 11:23:37.609796 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:37.609730 2556 generic.go:356] "Generic (PLEG): container finished" podID="4977506f-95a4-476c-80f6-8664a52d7f15" containerID="488cc8fe175187f61fdffaf9574f0645668b98cf6bca86fc7c9920db3742012f" exitCode=0 Sep 14 11:23:37.610239 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:37.609802 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39ab34-skcgr" event={"ID":"4977506f-95a4-476c-80f6-8664a52d7f15","Type":"ContainerDied","Data":"488cc8fe175187f61fdffaf9574f0645668b98cf6bca86fc7c9920db3742012f"} Sep 14 11:23:37.610239 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:37.609864 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39ab34-skcgr" event={"ID":"4977506f-95a4-476c-80f6-8664a52d7f15","Type":"ContainerStarted","Data":"469be408c0730343b7c8200be252d21d50beb51feddfe2e607d2ceae5bed05e6"} Sep 14 11:23:37.611315 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:37.611291 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj" event={"ID":"6327e7cc-7014-4b42-9207-1ff80b041bca","Type":"ContainerStarted","Data":"262cb49ec899d172b67016d6502e29bbf8843392533b8907b346fb7a40b3cc7f"} Sep 14 11:23:38.927789 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:38.927765 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39ab34-skcgr" Sep 14 11:23:39.003580 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:39.003559 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4977506f-95a4-476c-80f6-8664a52d7f15-maas-api-service-ca\") pod \"4977506f-95a4-476c-80f6-8664a52d7f15\" (UID: \"4977506f-95a4-476c-80f6-8664a52d7f15\") " Sep 14 11:23:39.003932 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:39.003901 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4977506f-95a4-476c-80f6-8664a52d7f15-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4977506f-95a4-476c-80f6-8664a52d7f15" (UID: "4977506f-95a4-476c-80f6-8664a52d7f15"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:23:39.104280 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:39.104256 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4977506f-95a4-476c-80f6-8664a52d7f15-maas-api-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:39.624019 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:39.623985 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39ab34-skcgr" Sep 14 11:23:39.624224 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:39.624022 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39ab34-skcgr" event={"ID":"4977506f-95a4-476c-80f6-8664a52d7f15","Type":"ContainerDied","Data":"469be408c0730343b7c8200be252d21d50beb51feddfe2e607d2ceae5bed05e6"} Sep 14 11:23:39.624224 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:39.624080 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="469be408c0730343b7c8200be252d21d50beb51feddfe2e607d2ceae5bed05e6" Sep 14 11:23:39.625682 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:39.625655 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj" event={"ID":"6327e7cc-7014-4b42-9207-1ff80b041bca","Type":"ContainerStarted","Data":"50544dda66b2a408d9471f1d0d30fb082fdc7c284494b48636f66c5c2efada9a"} Sep 14 11:23:39.625809 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:39.625789 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj" Sep 14 11:23:39.659538 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:39.659493 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj" podStartSLOduration=1.437755574 podStartE2EDuration="3.659480685s" totalImagesPullingTime="2.221725111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:23:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:23:39.658009535 +0000 UTC m=+1377.721412445" watchObservedRunningTime="2026-09-14 11:23:39.659480685 +0000 UTC m=+1377.722883596" Sep 14 11:23:41.103202 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.103178 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:23:41.122045 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.121996 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nl5b9\" (UniqueName: \"kubernetes.io/projected/9f43dd7d-23a3-4749-a795-562f8a28497b-kube-api-access-nl5b9\") pod \"9f43dd7d-23a3-4749-a795-562f8a28497b\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " Sep 14 11:23:41.122170 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.122043 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-istio-data\") pod \"9f43dd7d-23a3-4749-a795-562f8a28497b\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " Sep 14 11:23:41.122170 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.122064 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f43dd7d-23a3-4749-a795-562f8a28497b-istiod-ca-cert\") pod \"9f43dd7d-23a3-4749-a795-562f8a28497b\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " Sep 14 11:23:41.122170 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.122124 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-istio-envoy\") pod \"9f43dd7d-23a3-4749-a795-562f8a28497b\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " Sep 14 11:23:41.122330 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.122172 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f43dd7d-23a3-4749-a795-562f8a28497b-istio-token\") pod \"9f43dd7d-23a3-4749-a795-562f8a28497b\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " Sep 14 11:23:41.122330 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.122190 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-workload-certs\") pod \"9f43dd7d-23a3-4749-a795-562f8a28497b\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " Sep 14 11:23:41.122330 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.122219 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-workload-socket\") pod \"9f43dd7d-23a3-4749-a795-562f8a28497b\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " Sep 14 11:23:41.122330 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.122268 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-credential-socket\") pod \"9f43dd7d-23a3-4749-a795-562f8a28497b\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " Sep 14 11:23:41.122330 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.122304 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f43dd7d-23a3-4749-a795-562f8a28497b-istio-podinfo\") pod \"9f43dd7d-23a3-4749-a795-562f8a28497b\" (UID: \"9f43dd7d-23a3-4749-a795-562f8a28497b\") " Sep 14 11:23:41.122584 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.122337 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-istio-data" (OuterVolumeSpecName: "istio-data") pod "9f43dd7d-23a3-4749-a795-562f8a28497b" (UID: "9f43dd7d-23a3-4749-a795-562f8a28497b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:23:41.122584 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.122438 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9f43dd7d-23a3-4749-a795-562f8a28497b" (UID: "9f43dd7d-23a3-4749-a795-562f8a28497b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:23:41.122584 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.122481 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f43dd7d-23a3-4749-a795-562f8a28497b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9f43dd7d-23a3-4749-a795-562f8a28497b" (UID: "9f43dd7d-23a3-4749-a795-562f8a28497b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:23:41.122584 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.122576 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9f43dd7d-23a3-4749-a795-562f8a28497b" (UID: "9f43dd7d-23a3-4749-a795-562f8a28497b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:23:41.122773 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.122688 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9f43dd7d-23a3-4749-a795-562f8a28497b" (UID: "9f43dd7d-23a3-4749-a795-562f8a28497b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:23:41.122773 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.122703 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:41.122773 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.122724 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:41.122773 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.122737 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:41.122773 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.122751 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f43dd7d-23a3-4749-a795-562f8a28497b-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:41.124320 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.124298 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9f43dd7d-23a3-4749-a795-562f8a28497b" (UID: "9f43dd7d-23a3-4749-a795-562f8a28497b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:23:41.124390 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.124340 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f43dd7d-23a3-4749-a795-562f8a28497b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9f43dd7d-23a3-4749-a795-562f8a28497b" (UID: "9f43dd7d-23a3-4749-a795-562f8a28497b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 11:23:41.124517 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.124494 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f43dd7d-23a3-4749-a795-562f8a28497b-istio-token" (OuterVolumeSpecName: "istio-token") pod "9f43dd7d-23a3-4749-a795-562f8a28497b" (UID: "9f43dd7d-23a3-4749-a795-562f8a28497b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:23:41.124554 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.124527 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f43dd7d-23a3-4749-a795-562f8a28497b-kube-api-access-nl5b9" (OuterVolumeSpecName: "kube-api-access-nl5b9") pod "9f43dd7d-23a3-4749-a795-562f8a28497b" (UID: "9f43dd7d-23a3-4749-a795-562f8a28497b"). InnerVolumeSpecName "kube-api-access-nl5b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:23:41.223329 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.223308 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:41.223329 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.223329 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f43dd7d-23a3-4749-a795-562f8a28497b-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:41.223487 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.223338 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f43dd7d-23a3-4749-a795-562f8a28497b-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:41.223487 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.223347 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f43dd7d-23a3-4749-a795-562f8a28497b-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:41.223487 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.223355 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl5b9\" (UniqueName: \"kubernetes.io/projected/9f43dd7d-23a3-4749-a795-562f8a28497b-kube-api-access-nl5b9\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:23:41.637373 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.637341 2556 generic.go:356] "Generic (PLEG): container finished" podID="9f43dd7d-23a3-4749-a795-562f8a28497b" containerID="c5d2f86ecca952cd91bd4209f41ce2ec7020fee9080eaae7d53c2ca8a04a66f1" exitCode=0 Sep 14 11:23:41.637535 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.637390 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" Sep 14 11:23:41.637535 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.637432 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" event={"ID":"9f43dd7d-23a3-4749-a795-562f8a28497b","Type":"ContainerDied","Data":"c5d2f86ecca952cd91bd4209f41ce2ec7020fee9080eaae7d53c2ca8a04a66f1"} Sep 14 11:23:41.637535 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.637493 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr" event={"ID":"9f43dd7d-23a3-4749-a795-562f8a28497b","Type":"ContainerDied","Data":"ddc5b2db624689dd5901f52aae6d39a7ab69e29c7a46dc5b9084091a0c536421"} Sep 14 11:23:41.637710 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.637538 2556 scope.go:117] "RemoveContainer" containerID="c5d2f86ecca952cd91bd4209f41ce2ec7020fee9080eaae7d53c2ca8a04a66f1" Sep 14 11:23:41.646789 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.646763 2556 scope.go:117] "RemoveContainer" containerID="7006ec3d7f921d10d9dba6ee22d646ec37c25132e8e74e9bb36d010e552a4005" Sep 14 11:23:41.655454 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.655440 2556 scope.go:117] "RemoveContainer" containerID="c5d2f86ecca952cd91bd4209f41ce2ec7020fee9080eaae7d53c2ca8a04a66f1" Sep 14 11:23:41.655703 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:23:41.655684 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5d2f86ecca952cd91bd4209f41ce2ec7020fee9080eaae7d53c2ca8a04a66f1\": container with ID starting with c5d2f86ecca952cd91bd4209f41ce2ec7020fee9080eaae7d53c2ca8a04a66f1 not found: ID does not exist" containerID="c5d2f86ecca952cd91bd4209f41ce2ec7020fee9080eaae7d53c2ca8a04a66f1" Sep 14 11:23:41.655754 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.655710 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5d2f86ecca952cd91bd4209f41ce2ec7020fee9080eaae7d53c2ca8a04a66f1"} err="failed to get container status \"c5d2f86ecca952cd91bd4209f41ce2ec7020fee9080eaae7d53c2ca8a04a66f1\": rpc error: code = NotFound desc = could not find container \"c5d2f86ecca952cd91bd4209f41ce2ec7020fee9080eaae7d53c2ca8a04a66f1\": container with ID starting with c5d2f86ecca952cd91bd4209f41ce2ec7020fee9080eaae7d53c2ca8a04a66f1 not found: ID does not exist" Sep 14 11:23:41.655754 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.655724 2556 scope.go:117] "RemoveContainer" containerID="7006ec3d7f921d10d9dba6ee22d646ec37c25132e8e74e9bb36d010e552a4005" Sep 14 11:23:41.655981 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:23:41.655963 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7006ec3d7f921d10d9dba6ee22d646ec37c25132e8e74e9bb36d010e552a4005\": container with ID starting with 7006ec3d7f921d10d9dba6ee22d646ec37c25132e8e74e9bb36d010e552a4005 not found: ID does not exist" containerID="7006ec3d7f921d10d9dba6ee22d646ec37c25132e8e74e9bb36d010e552a4005" Sep 14 11:23:41.656025 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.655985 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7006ec3d7f921d10d9dba6ee22d646ec37c25132e8e74e9bb36d010e552a4005"} err="failed to get container status \"7006ec3d7f921d10d9dba6ee22d646ec37c25132e8e74e9bb36d010e552a4005\": rpc error: code = NotFound desc = could not find container \"7006ec3d7f921d10d9dba6ee22d646ec37c25132e8e74e9bb36d010e552a4005\": container with ID starting with 7006ec3d7f921d10d9dba6ee22d646ec37c25132e8e74e9bb36d010e552a4005 not found: ID does not exist" Sep 14 11:23:41.661156 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.661132 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr"] Sep 14 11:23:41.666888 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:41.666869 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6d272c-openshift-default-664498d46-dw4hr"] Sep 14 11:23:42.390690 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:42.390656 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f43dd7d-23a3-4749-a795-562f8a28497b" path="/var/lib/kubelet/pods/9f43dd7d-23a3-4749-a795-562f8a28497b/volumes" Sep 14 11:23:42.506728 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:42.506703 2556 scope.go:117] "RemoveContainer" containerID="f73e060faad9608cf8553deaa6bdf3fb02aab411f6434c8682c1a3a079f7b441" Sep 14 11:23:42.514510 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:42.514487 2556 scope.go:117] "RemoveContainer" containerID="2565ad8f2ebaa96ee725699dcfb3467b9695dfa757694ce9a1320da030f13a72" Sep 14 11:23:47.525162 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:47.525126 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39ab34-skcgr"] Sep 14 11:23:47.528871 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:47.528844 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39ab34-skcgr"] Sep 14 11:23:48.382894 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:48.382853 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4977506f-95a4-476c-80f6-8664a52d7f15" path="/var/lib/kubelet/pods/4977506f-95a4-476c-80f6-8664a52d7f15/volumes" Sep 14 11:23:50.632953 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:50.632874 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj" Sep 14 11:23:53.704681 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:53.704652 2556 generic.go:356] "Generic (PLEG): container finished" podID="ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6" containerID="de5c3cf8e282799d90e34a35bcfb42e32c9a68ecd903bf8af3ad3935c6b09f2c" exitCode=0 Sep 14 11:23:53.705133 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:53.704734 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" event={"ID":"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6","Type":"ContainerDied","Data":"de5c3cf8e282799d90e34a35bcfb42e32c9a68ecd903bf8af3ad3935c6b09f2c"} Sep 14 11:23:53.705214 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:53.705144 2556 scope.go:117] "RemoveContainer" containerID="de5c3cf8e282799d90e34a35bcfb42e32c9a68ecd903bf8af3ad3935c6b09f2c" Sep 14 11:23:53.706192 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:53.706156 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:23:53.706312 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:53.706233 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:23:53.706312 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:53.706281 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:23:54.714849 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:54.714812 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" event={"ID":"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6","Type":"ContainerStarted","Data":"a323e8d538a2ebea144f7b1b1fa563152a3f83cbd33fb2c1c4ae6becef53423c"} Sep 14 11:23:54.801192 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:54.801157 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv"] Sep 14 11:23:54.801687 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:54.801655 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" podUID="0376711b-9ae6-441c-8083-bf6086d85862" containerName="istio-proxy" containerID="cri-o://0333120bed9ae7af0196867688767c0b79012e8ba95361397c7db974e826746b" gracePeriod=30 Sep 14 11:23:55.575492 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.575458 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:55.575657 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.575621 2556 patch_prober.go:28] interesting pod/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 14 11:23:55.575701 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.575665 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" podUID="ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 14 11:23:55.708685 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.708648 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8"] Sep 14 11:23:55.714257 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.714233 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f43dd7d-23a3-4749-a795-562f8a28497b" containerName="istio-proxy" Sep 14 11:23:55.714426 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.714411 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f43dd7d-23a3-4749-a795-562f8a28497b" containerName="istio-proxy" Sep 14 11:23:55.714547 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.714534 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4977506f-95a4-476c-80f6-8664a52d7f15" containerName="revoke-keys" Sep 14 11:23:55.714637 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.714625 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4977506f-95a4-476c-80f6-8664a52d7f15" containerName="revoke-keys" Sep 14 11:23:55.714732 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.714721 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f43dd7d-23a3-4749-a795-562f8a28497b" containerName="istio-proxy" Sep 14 11:23:55.714818 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.714806 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f43dd7d-23a3-4749-a795-562f8a28497b" containerName="istio-proxy" Sep 14 11:23:55.715000 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.714987 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f43dd7d-23a3-4749-a795-562f8a28497b" containerName="istio-proxy" Sep 14 11:23:55.715423 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.715408 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4977506f-95a4-476c-80f6-8664a52d7f15" containerName="revoke-keys" Sep 14 11:23:55.715507 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.715497 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f43dd7d-23a3-4749-a795-562f8a28497b" containerName="istio-proxy" Sep 14 11:23:55.717996 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.717978 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.722082 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.722061 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-55c025-openshift-default-dockercfg-l5qgw\"" Sep 14 11:23:55.725035 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.725016 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8"] Sep 14 11:23:55.802112 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.794453 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf338544-9d39-4e9e-9668-ba204a6e2fde-istio-podinfo\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.802112 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.794726 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-istio-data\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.802112 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.794767 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-credential-socket\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.802112 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.794822 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-workload-socket\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.802112 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.794880 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-istio-envoy\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.802112 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.794946 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-workload-certs\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.802112 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.794975 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf338544-9d39-4e9e-9668-ba204a6e2fde-istiod-ca-cert\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.802112 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.795016 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrhnw\" (UniqueName: \"kubernetes.io/projected/cf338544-9d39-4e9e-9668-ba204a6e2fde-kube-api-access-jrhnw\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.802112 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.795062 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf338544-9d39-4e9e-9668-ba204a6e2fde-istio-token\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.896111 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.896052 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-istio-data\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.896327 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.896134 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-credential-socket\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.896327 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.896199 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-workload-socket\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.896327 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.896258 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-istio-envoy\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.896502 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.896337 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-workload-certs\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.896502 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.896360 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf338544-9d39-4e9e-9668-ba204a6e2fde-istiod-ca-cert\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.896502 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.896392 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrhnw\" (UniqueName: \"kubernetes.io/projected/cf338544-9d39-4e9e-9668-ba204a6e2fde-kube-api-access-jrhnw\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.896502 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.896422 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf338544-9d39-4e9e-9668-ba204a6e2fde-istio-token\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.896686 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.896509 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf338544-9d39-4e9e-9668-ba204a6e2fde-istio-podinfo\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.897998 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.897966 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf338544-9d39-4e9e-9668-ba204a6e2fde-istiod-ca-cert\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.898561 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.898536 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-workload-certs\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.899014 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.898982 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-istio-data\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.899276 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.899255 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-credential-socket\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.901256 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.899884 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-workload-socket\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.904317 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.904290 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-istio-envoy\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.913165 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.913145 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf338544-9d39-4e9e-9668-ba204a6e2fde-istio-podinfo\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.915154 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.915132 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf338544-9d39-4e9e-9668-ba204a6e2fde-istio-token\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:55.917200 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:55.917158 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrhnw\" (UniqueName: \"kubernetes.io/projected/cf338544-9d39-4e9e-9668-ba204a6e2fde-kube-api-access-jrhnw\") pod \"e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:56.030616 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:56.030580 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:56.330199 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:56.328832 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8"] Sep 14 11:23:56.333321 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:56.333285 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:23:56.333419 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:56.333350 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:23:56.333419 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:56.333380 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:23:56.576756 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:56.576673 2556 patch_prober.go:28] interesting pod/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 14 11:23:56.576756 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:56.576740 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" podUID="ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 14 11:23:56.727130 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:56.727072 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" event={"ID":"cf338544-9d39-4e9e-9668-ba204a6e2fde","Type":"ContainerStarted","Data":"c3b1e079d480f6c5be8d4406a687d93be3d88adf6339e29c175cae718a1b46d1"} Sep 14 11:23:56.727566 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:56.727143 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" event={"ID":"cf338544-9d39-4e9e-9668-ba204a6e2fde","Type":"ContainerStarted","Data":"23d180ec6546afa957f529954517aee388981dfc7c3f7f7ed0ddecf8b6b05f1d"} Sep 14 11:23:56.754914 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:56.754866 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" podStartSLOduration=1.754851749 podStartE2EDuration="1.754851749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:23:56.75254944 +0000 UTC m=+1394.815952349" watchObservedRunningTime="2026-09-14 11:23:56.754851749 +0000 UTC m=+1394.818254658" Sep 14 11:23:57.031106 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:57.031059 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:57.035818 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:57.035795 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:57.576376 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:57.576342 2556 patch_prober.go:28] interesting pod/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 14 11:23:57.576560 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:57.576393 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" podUID="ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 14 11:23:57.731534 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:57.731503 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:57.732365 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:57.732344 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:23:58.575805 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:58.575769 2556 patch_prober.go:28] interesting pod/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 14 11:23:58.575971 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:58.575823 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" podUID="ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 14 11:23:59.582873 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:59.582838 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:59.583367 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:59.583204 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:23:59.584638 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:23:59.584618 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:24:00.042205 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.042183 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:24:00.149290 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.149263 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-workload-socket\") pod \"0376711b-9ae6-441c-8083-bf6086d85862\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " Sep 14 11:24:00.149462 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.149319 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-istio-data\") pod \"0376711b-9ae6-441c-8083-bf6086d85862\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " Sep 14 11:24:00.149462 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.149355 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0376711b-9ae6-441c-8083-bf6086d85862-istiod-ca-cert\") pod \"0376711b-9ae6-441c-8083-bf6086d85862\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " Sep 14 11:24:00.149462 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.149369 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-istio-envoy\") pod \"0376711b-9ae6-441c-8083-bf6086d85862\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " Sep 14 11:24:00.149462 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.149404 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0376711b-9ae6-441c-8083-bf6086d85862-istio-token\") pod \"0376711b-9ae6-441c-8083-bf6086d85862\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " Sep 14 11:24:00.149462 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.149439 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-credential-socket\") pod \"0376711b-9ae6-441c-8083-bf6086d85862\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " Sep 14 11:24:00.149714 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.149493 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0376711b-9ae6-441c-8083-bf6086d85862-istio-podinfo\") pod \"0376711b-9ae6-441c-8083-bf6086d85862\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " Sep 14 11:24:00.149714 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.149517 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-workload-certs\") pod \"0376711b-9ae6-441c-8083-bf6086d85862\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " Sep 14 11:24:00.149714 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.149539 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j56rr\" (UniqueName: \"kubernetes.io/projected/0376711b-9ae6-441c-8083-bf6086d85862-kube-api-access-j56rr\") pod \"0376711b-9ae6-441c-8083-bf6086d85862\" (UID: \"0376711b-9ae6-441c-8083-bf6086d85862\") " Sep 14 11:24:00.149714 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.149548 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0376711b-9ae6-441c-8083-bf6086d85862" (UID: "0376711b-9ae6-441c-8083-bf6086d85862"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:24:00.149714 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.149635 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-istio-data" (OuterVolumeSpecName: "istio-data") pod "0376711b-9ae6-441c-8083-bf6086d85862" (UID: "0376711b-9ae6-441c-8083-bf6086d85862"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:24:00.149972 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.149766 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0376711b-9ae6-441c-8083-bf6086d85862" (UID: "0376711b-9ae6-441c-8083-bf6086d85862"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:24:00.149972 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.149784 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0376711b-9ae6-441c-8083-bf6086d85862" (UID: "0376711b-9ae6-441c-8083-bf6086d85862"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:24:00.149972 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.149798 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0376711b-9ae6-441c-8083-bf6086d85862-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0376711b-9ae6-441c-8083-bf6086d85862" (UID: "0376711b-9ae6-441c-8083-bf6086d85862"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:24:00.149972 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.149923 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:24:00.149972 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.149937 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:24:00.149972 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.149947 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:24:00.149972 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.149960 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0376711b-9ae6-441c-8083-bf6086d85862-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:24:00.150269 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.149977 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:24:00.151527 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.151506 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0376711b-9ae6-441c-8083-bf6086d85862" (UID: "0376711b-9ae6-441c-8083-bf6086d85862"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:24:00.151685 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.151668 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0376711b-9ae6-441c-8083-bf6086d85862-istio-token" (OuterVolumeSpecName: "istio-token") pod "0376711b-9ae6-441c-8083-bf6086d85862" (UID: "0376711b-9ae6-441c-8083-bf6086d85862"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:24:00.151883 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.151869 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0376711b-9ae6-441c-8083-bf6086d85862-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0376711b-9ae6-441c-8083-bf6086d85862" (UID: "0376711b-9ae6-441c-8083-bf6086d85862"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 11:24:00.151934 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.151912 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0376711b-9ae6-441c-8083-bf6086d85862-kube-api-access-j56rr" (OuterVolumeSpecName: "kube-api-access-j56rr") pod "0376711b-9ae6-441c-8083-bf6086d85862" (UID: "0376711b-9ae6-441c-8083-bf6086d85862"). InnerVolumeSpecName "kube-api-access-j56rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:24:00.250788 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.250762 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0376711b-9ae6-441c-8083-bf6086d85862-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:24:00.250788 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.250786 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0376711b-9ae6-441c-8083-bf6086d85862-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:24:00.250930 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.250794 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0376711b-9ae6-441c-8083-bf6086d85862-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:24:00.250930 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.250803 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j56rr\" (UniqueName: \"kubernetes.io/projected/0376711b-9ae6-441c-8083-bf6086d85862-kube-api-access-j56rr\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:24:00.749036 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.749004 2556 generic.go:356] "Generic (PLEG): container finished" podID="0376711b-9ae6-441c-8083-bf6086d85862" containerID="0333120bed9ae7af0196867688767c0b79012e8ba95361397c7db974e826746b" exitCode=0 Sep 14 11:24:00.749427 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.749062 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" Sep 14 11:24:00.749427 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.749114 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" event={"ID":"0376711b-9ae6-441c-8083-bf6086d85862","Type":"ContainerDied","Data":"0333120bed9ae7af0196867688767c0b79012e8ba95361397c7db974e826746b"} Sep 14 11:24:00.749427 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.749205 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv" event={"ID":"0376711b-9ae6-441c-8083-bf6086d85862","Type":"ContainerDied","Data":"a257d8121793002187a89d0013ed10fa149d34b43e70d55e750dca041718d6e5"} Sep 14 11:24:00.749427 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.749234 2556 scope.go:117] "RemoveContainer" containerID="0333120bed9ae7af0196867688767c0b79012e8ba95361397c7db974e826746b" Sep 14 11:24:00.757973 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.757956 2556 scope.go:117] "RemoveContainer" containerID="02b41bc73bd9922b833a9dd9ecc282df26b728a7c95764b6c351f864804e946f" Sep 14 11:24:00.765849 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.765833 2556 scope.go:117] "RemoveContainer" containerID="0333120bed9ae7af0196867688767c0b79012e8ba95361397c7db974e826746b" Sep 14 11:24:00.766113 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:24:00.766082 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0333120bed9ae7af0196867688767c0b79012e8ba95361397c7db974e826746b\": container with ID starting with 0333120bed9ae7af0196867688767c0b79012e8ba95361397c7db974e826746b not found: ID does not exist" containerID="0333120bed9ae7af0196867688767c0b79012e8ba95361397c7db974e826746b" Sep 14 11:24:00.766158 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.766122 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0333120bed9ae7af0196867688767c0b79012e8ba95361397c7db974e826746b"} err="failed to get container status \"0333120bed9ae7af0196867688767c0b79012e8ba95361397c7db974e826746b\": rpc error: code = NotFound desc = could not find container \"0333120bed9ae7af0196867688767c0b79012e8ba95361397c7db974e826746b\": container with ID starting with 0333120bed9ae7af0196867688767c0b79012e8ba95361397c7db974e826746b not found: ID does not exist" Sep 14 11:24:00.766158 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.766140 2556 scope.go:117] "RemoveContainer" containerID="02b41bc73bd9922b833a9dd9ecc282df26b728a7c95764b6c351f864804e946f" Sep 14 11:24:00.766390 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:24:00.766373 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02b41bc73bd9922b833a9dd9ecc282df26b728a7c95764b6c351f864804e946f\": container with ID starting with 02b41bc73bd9922b833a9dd9ecc282df26b728a7c95764b6c351f864804e946f not found: ID does not exist" containerID="02b41bc73bd9922b833a9dd9ecc282df26b728a7c95764b6c351f864804e946f" Sep 14 11:24:00.766429 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.766396 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02b41bc73bd9922b833a9dd9ecc282df26b728a7c95764b6c351f864804e946f"} err="failed to get container status \"02b41bc73bd9922b833a9dd9ecc282df26b728a7c95764b6c351f864804e946f\": rpc error: code = NotFound desc = could not find container \"02b41bc73bd9922b833a9dd9ecc282df26b728a7c95764b6c351f864804e946f\": container with ID starting with 02b41bc73bd9922b833a9dd9ecc282df26b728a7c95764b6c351f864804e946f not found: ID does not exist" Sep 14 11:24:00.786566 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.786541 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv"] Sep 14 11:24:00.795277 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:00.795258 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-39ab34-openshift-default-5f8fd65bb4-72zxv"] Sep 14 11:24:02.382358 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:02.382329 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0376711b-9ae6-441c-8083-bf6086d85862" path="/var/lib/kubelet/pods/0376711b-9ae6-441c-8083-bf6086d85862/volumes" Sep 14 11:24:02.820364 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:02.820291 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx"] Sep 14 11:24:02.820636 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:02.820625 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0376711b-9ae6-441c-8083-bf6086d85862" containerName="istio-proxy" Sep 14 11:24:02.820678 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:02.820638 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0376711b-9ae6-441c-8083-bf6086d85862" containerName="istio-proxy" Sep 14 11:24:02.820678 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:02.820650 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0376711b-9ae6-441c-8083-bf6086d85862" containerName="istio-proxy" Sep 14 11:24:02.820678 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:02.820655 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0376711b-9ae6-441c-8083-bf6086d85862" containerName="istio-proxy" Sep 14 11:24:02.820774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:02.820706 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0376711b-9ae6-441c-8083-bf6086d85862" containerName="istio-proxy" Sep 14 11:24:02.823656 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:02.823638 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx" Sep 14 11:24:02.826866 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:02.826843 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-55c025-dockercfg-kgnk5\"" Sep 14 11:24:02.827388 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:02.827373 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-55c025\"" Sep 14 11:24:02.837590 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:02.837570 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx"] Sep 14 11:24:02.976175 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:02.976143 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8x52\" (UniqueName: \"kubernetes.io/projected/45ec7d33-3016-4f95-91a3-1fe8e039727f-kube-api-access-q8x52\") pod \"payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx\" (UID: \"45ec7d33-3016-4f95-91a3-1fe8e039727f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx" Sep 14 11:24:02.976345 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:02.976201 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ec7d33-3016-4f95-91a3-1fe8e039727f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx\" (UID: \"45ec7d33-3016-4f95-91a3-1fe8e039727f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx" Sep 14 11:24:03.077189 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:03.077115 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8x52\" (UniqueName: \"kubernetes.io/projected/45ec7d33-3016-4f95-91a3-1fe8e039727f-kube-api-access-q8x52\") pod \"payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx\" (UID: \"45ec7d33-3016-4f95-91a3-1fe8e039727f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx" Sep 14 11:24:03.077189 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:03.077170 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ec7d33-3016-4f95-91a3-1fe8e039727f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx\" (UID: \"45ec7d33-3016-4f95-91a3-1fe8e039727f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx" Sep 14 11:24:03.077835 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:03.077816 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ec7d33-3016-4f95-91a3-1fe8e039727f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx\" (UID: \"45ec7d33-3016-4f95-91a3-1fe8e039727f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx" Sep 14 11:24:03.086568 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:03.086543 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8x52\" (UniqueName: \"kubernetes.io/projected/45ec7d33-3016-4f95-91a3-1fe8e039727f-kube-api-access-q8x52\") pod \"payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx\" (UID: \"45ec7d33-3016-4f95-91a3-1fe8e039727f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx" Sep 14 11:24:03.133257 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:03.133230 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx" Sep 14 11:24:03.404393 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:03.404372 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx"] Sep 14 11:24:03.406072 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:24:03.406044 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ec7d33_3016_4f95_91a3_1fe8e039727f.slice/crio-52d197d029bc3ce725eaa3b49d2ec6129e949ad14aa8b04c5ea9c3c2e9ef261e WatchSource:0}: Error finding container 52d197d029bc3ce725eaa3b49d2ec6129e949ad14aa8b04c5ea9c3c2e9ef261e: Status 404 returned error can't find the container with id 52d197d029bc3ce725eaa3b49d2ec6129e949ad14aa8b04c5ea9c3c2e9ef261e Sep 14 11:24:03.765471 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:03.765394 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx" event={"ID":"45ec7d33-3016-4f95-91a3-1fe8e039727f","Type":"ContainerStarted","Data":"f4ff6363117c18f12fba6157a004943ab7273146e36a67bc0c2a6a76c8ee7026"} Sep 14 11:24:03.765471 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:03.765448 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx" event={"ID":"45ec7d33-3016-4f95-91a3-1fe8e039727f","Type":"ContainerStarted","Data":"52d197d029bc3ce725eaa3b49d2ec6129e949ad14aa8b04c5ea9c3c2e9ef261e"} Sep 14 11:24:03.765642 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:03.765519 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx" Sep 14 11:24:03.788967 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:03.788915 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx" podStartSLOduration=1.788896978 podStartE2EDuration="1.788896978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:24:03.787942625 +0000 UTC m=+1401.851345537" watchObservedRunningTime="2026-09-14 11:24:03.788896978 +0000 UTC m=+1401.852299889" Sep 14 11:24:05.237496 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:05.237462 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw"] Sep 14 11:24:05.237892 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:05.237879 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0376711b-9ae6-441c-8083-bf6086d85862" containerName="istio-proxy" Sep 14 11:24:05.239819 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:05.239804 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:24:05.242599 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:05.242579 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6f3d0f\"/\"distinct-2-w0-6f3d0f-kserve-self-signed-certs\"" Sep 14 11:24:05.250630 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:05.250609 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw"] Sep 14 11:24:05.396058 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:05.396024 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1feb9dd5-e808-4402-a88f-6c5726560ac4-tls-certs\") pod \"distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw\" (UID: \"1feb9dd5-e808-4402-a88f-6c5726560ac4\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:24:05.396224 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:05.396078 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1feb9dd5-e808-4402-a88f-6c5726560ac4-model-cache\") pod \"distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw\" (UID: \"1feb9dd5-e808-4402-a88f-6c5726560ac4\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:24:05.396224 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:05.396166 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1feb9dd5-e808-4402-a88f-6c5726560ac4-dshm\") pod \"distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw\" (UID: \"1feb9dd5-e808-4402-a88f-6c5726560ac4\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:24:05.396224 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:05.396191 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1feb9dd5-e808-4402-a88f-6c5726560ac4-home\") pod \"distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw\" (UID: \"1feb9dd5-e808-4402-a88f-6c5726560ac4\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:24:05.396354 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:05.396222 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq4xx\" (UniqueName: \"kubernetes.io/projected/1feb9dd5-e808-4402-a88f-6c5726560ac4-kube-api-access-pq4xx\") pod \"distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw\" (UID: \"1feb9dd5-e808-4402-a88f-6c5726560ac4\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:24:05.396354 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:05.396256 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1feb9dd5-e808-4402-a88f-6c5726560ac4-tmp-dir\") pod \"distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw\" (UID: \"1feb9dd5-e808-4402-a88f-6c5726560ac4\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:24:05.497435 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:05.497375 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1feb9dd5-e808-4402-a88f-6c5726560ac4-dshm\") pod \"distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw\" (UID: \"1feb9dd5-e808-4402-a88f-6c5726560ac4\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:24:05.497435 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:05.497403 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1feb9dd5-e808-4402-a88f-6c5726560ac4-home\") pod \"distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw\" (UID: \"1feb9dd5-e808-4402-a88f-6c5726560ac4\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:24:05.497435 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:05.497436 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq4xx\" (UniqueName: \"kubernetes.io/projected/1feb9dd5-e808-4402-a88f-6c5726560ac4-kube-api-access-pq4xx\") pod \"distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw\" (UID: \"1feb9dd5-e808-4402-a88f-6c5726560ac4\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:24:05.497675 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:05.497460 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1feb9dd5-e808-4402-a88f-6c5726560ac4-tmp-dir\") pod \"distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw\" (UID: \"1feb9dd5-e808-4402-a88f-6c5726560ac4\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:24:05.497675 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:05.497545 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1feb9dd5-e808-4402-a88f-6c5726560ac4-tls-certs\") pod \"distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw\" (UID: \"1feb9dd5-e808-4402-a88f-6c5726560ac4\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:24:05.497675 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:05.497590 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1feb9dd5-e808-4402-a88f-6c5726560ac4-model-cache\") pod \"distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw\" (UID: \"1feb9dd5-e808-4402-a88f-6c5726560ac4\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:24:05.497839 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:05.497796 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1feb9dd5-e808-4402-a88f-6c5726560ac4-home\") pod \"distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw\" (UID: \"1feb9dd5-e808-4402-a88f-6c5726560ac4\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:24:05.497907 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:05.497889 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1feb9dd5-e808-4402-a88f-6c5726560ac4-tmp-dir\") pod \"distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw\" (UID: \"1feb9dd5-e808-4402-a88f-6c5726560ac4\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:24:05.497975 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:05.497958 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1feb9dd5-e808-4402-a88f-6c5726560ac4-model-cache\") pod \"distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw\" (UID: \"1feb9dd5-e808-4402-a88f-6c5726560ac4\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:24:05.499674 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:05.499655 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1feb9dd5-e808-4402-a88f-6c5726560ac4-dshm\") pod \"distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw\" (UID: \"1feb9dd5-e808-4402-a88f-6c5726560ac4\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:24:05.499905 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:05.499889 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1feb9dd5-e808-4402-a88f-6c5726560ac4-tls-certs\") pod \"distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw\" (UID: \"1feb9dd5-e808-4402-a88f-6c5726560ac4\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:24:05.506457 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:05.506435 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq4xx\" (UniqueName: \"kubernetes.io/projected/1feb9dd5-e808-4402-a88f-6c5726560ac4-kube-api-access-pq4xx\") pod \"distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw\" (UID: \"1feb9dd5-e808-4402-a88f-6c5726560ac4\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:24:05.549247 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:05.549227 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:24:05.826064 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:05.826036 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw"] Sep 14 11:24:05.827712 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:24:05.827684 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1feb9dd5_e808_4402_a88f_6c5726560ac4.slice/crio-3c2175fc6ba50a0d491230aeaa829e9536d9c1c9bf70ae485d40c44da12053b5 WatchSource:0}: Error finding container 3c2175fc6ba50a0d491230aeaa829e9536d9c1c9bf70ae485d40c44da12053b5: Status 404 returned error can't find the container with id 3c2175fc6ba50a0d491230aeaa829e9536d9c1c9bf70ae485d40c44da12053b5 Sep 14 11:24:06.789130 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:06.789060 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" event={"ID":"1feb9dd5-e808-4402-a88f-6c5726560ac4","Type":"ContainerStarted","Data":"01ae59f1fc273123bd58f635aa3a915bb3ae66b653cb59cbea5127704b0e1958"} Sep 14 11:24:06.789575 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:06.789144 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" event={"ID":"1feb9dd5-e808-4402-a88f-6c5726560ac4","Type":"ContainerStarted","Data":"3c2175fc6ba50a0d491230aeaa829e9536d9c1c9bf70ae485d40c44da12053b5"} Sep 14 11:24:06.818032 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:06.817988 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" podStartSLOduration=1.817975772 podStartE2EDuration="1.817975772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:24:06.814465546 +0000 UTC m=+1404.877868457" watchObservedRunningTime="2026-09-14 11:24:06.817975772 +0000 UTC m=+1404.881378682" Sep 14 11:24:14.777643 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:14.777616 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx" Sep 14 11:24:15.550262 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:15.550229 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:24:15.550448 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:15.550279 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:24:15.562102 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:15.562067 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:24:15.843428 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:15.843356 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:24:29.726224 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.726192 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p"] Sep 14 11:24:29.729880 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.729863 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.733134 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.733115 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-3f02a5-openshift-default-dockercfg-57tkl\"" Sep 14 11:24:29.771717 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.771681 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p"] Sep 14 11:24:29.822693 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.822655 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istiod-ca-cert\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.822882 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.822717 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-workload-certs\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.822882 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.822763 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istio-data\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.822882 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.822812 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-workload-socket\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.822882 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.822851 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istio-envoy\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.822882 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.822873 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istio-podinfo\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.823176 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.822914 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istio-token\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.823176 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.822991 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p286j\" (UniqueName: \"kubernetes.io/projected/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-kube-api-access-p286j\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.823176 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.823051 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-credential-socket\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.924235 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.924193 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-credential-socket\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.924403 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.924279 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istiod-ca-cert\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.924403 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.924365 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-workload-certs\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.924403 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.924392 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istio-data\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.924569 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.924428 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-workload-socket\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.924569 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.924465 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istio-envoy\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.924569 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.924486 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istio-podinfo\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.924569 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.924528 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istio-token\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.924758 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.924580 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p286j\" (UniqueName: \"kubernetes.io/projected/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-kube-api-access-p286j\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.924758 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.924588 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-credential-socket\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.924758 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.924734 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-workload-certs\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.924937 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.924911 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-workload-socket\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.924998 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.924952 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istiod-ca-cert\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.925134 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.925116 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istio-data\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.926821 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.926802 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istio-envoy\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.927131 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.927114 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istio-podinfo\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.936955 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.936928 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istio-token\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:29.939005 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:29.938983 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p286j\" (UniqueName: \"kubernetes.io/projected/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-kube-api-access-p286j\") pod \"e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:30.040741 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:30.040675 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:30.333339 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:30.333298 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p"] Sep 14 11:24:30.334686 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:24:30.334662 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f0f8a3_1879_470f_aedc_5a6dfb04e3bc.slice/crio-9dfbf2f47b2bf19f1df2640c42638701601968799e4fc7bb8d444af0dd21bb44 WatchSource:0}: Error finding container 9dfbf2f47b2bf19f1df2640c42638701601968799e4fc7bb8d444af0dd21bb44: Status 404 returned error can't find the container with id 9dfbf2f47b2bf19f1df2640c42638701601968799e4fc7bb8d444af0dd21bb44 Sep 14 11:24:30.337202 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:30.337172 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:24:30.337281 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:30.337233 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:24:30.337281 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:30.337260 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:24:30.920426 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:30.920393 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" event={"ID":"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc","Type":"ContainerStarted","Data":"f8ec02d95902757b3defae47a62f71116aebc0cef72a416ec34019b165815878"} Sep 14 11:24:30.921822 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:30.920444 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" event={"ID":"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc","Type":"ContainerStarted","Data":"9dfbf2f47b2bf19f1df2640c42638701601968799e4fc7bb8d444af0dd21bb44"} Sep 14 11:24:30.955989 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:30.955947 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" podStartSLOduration=1.9559315430000002 podStartE2EDuration="1.955931543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:24:30.952243603 +0000 UTC m=+1429.015646512" watchObservedRunningTime="2026-09-14 11:24:30.955931543 +0000 UTC m=+1429.019334452" Sep 14 11:24:31.041286 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:31.041267 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:31.046059 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:31.046038 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:31.930769 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:31.930729 2556 generic.go:356] "Generic (PLEG): container finished" podID="cf338544-9d39-4e9e-9668-ba204a6e2fde" containerID="c3b1e079d480f6c5be8d4406a687d93be3d88adf6339e29c175cae718a1b46d1" exitCode=0 Sep 14 11:24:31.931592 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:31.931559 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" event={"ID":"cf338544-9d39-4e9e-9668-ba204a6e2fde","Type":"ContainerDied","Data":"c3b1e079d480f6c5be8d4406a687d93be3d88adf6339e29c175cae718a1b46d1"} Sep 14 11:24:31.932008 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:31.931984 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:31.932177 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:31.932027 2556 scope.go:117] "RemoveContainer" containerID="c3b1e079d480f6c5be8d4406a687d93be3d88adf6339e29c175cae718a1b46d1" Sep 14 11:24:31.933862 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:31.933790 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:24:31.933960 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:31.933900 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:24:31.934027 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:31.933956 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:24:31.934560 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:31.934533 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:24:32.937953 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:32.937909 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" event={"ID":"cf338544-9d39-4e9e-9668-ba204a6e2fde","Type":"ContainerStarted","Data":"b8df8986d3a73c2abfb3fc1721a11a73b8935b8dac6693bbce9dbf4c8eea9e21"} Sep 14 11:24:33.030955 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:33.030919 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:24:33.031218 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:33.031191 2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 14 11:24:33.031307 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:33.031250 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" podUID="cf338544-9d39-4e9e-9668-ba204a6e2fde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 14 11:24:34.031620 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:34.031586 2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 14 11:24:34.032062 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:34.031657 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" podUID="cf338544-9d39-4e9e-9668-ba204a6e2fde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 14 11:24:35.031828 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:35.031749 2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 14 11:24:35.031828 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:35.031815 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" podUID="cf338544-9d39-4e9e-9668-ba204a6e2fde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 14 11:24:36.031406 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:36.031369 2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 14 11:24:36.031570 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:36.031448 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" podUID="cf338544-9d39-4e9e-9668-ba204a6e2fde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 14 11:24:37.031787 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:37.031748 2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 14 11:24:37.032277 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:37.031829 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" podUID="cf338544-9d39-4e9e-9668-ba204a6e2fde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 14 11:24:38.034806 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:38.034774 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:24:38.035265 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:38.035024 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:24:38.035595 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:38.035574 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:24:42.556584 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:42.556554 2556 scope.go:117] "RemoveContainer" containerID="f909b72ccbc7ead7f751cabe5245589597a338547a50e021181607c96862df54" Sep 14 11:24:42.564603 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:42.564582 2556 scope.go:117] "RemoveContainer" containerID="5f0ba376d65e91d8b247f3e743a1eaa46cf72f512f724b1bc0ff4aa9140aa04c" Sep 14 11:24:42.574784 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:24:42.574768 2556 scope.go:117] "RemoveContainer" containerID="8d2bab81a9bf61886aa13f0aab463c1739959a9e74ee0d9fe1af863aab3bc04b" Sep 14 11:25:06.112500 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:06.112463 2556 generic.go:356] "Generic (PLEG): container finished" podID="11f0f8a3-1879-470f-aedc-5a6dfb04e3bc" containerID="f8ec02d95902757b3defae47a62f71116aebc0cef72a416ec34019b165815878" exitCode=0 Sep 14 11:25:06.112911 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:06.112541 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" event={"ID":"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc","Type":"ContainerDied","Data":"f8ec02d95902757b3defae47a62f71116aebc0cef72a416ec34019b165815878"} Sep 14 11:25:06.112956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:06.112923 2556 scope.go:117] "RemoveContainer" containerID="f8ec02d95902757b3defae47a62f71116aebc0cef72a416ec34019b165815878" Sep 14 11:25:06.114050 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:06.114024 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:25:06.114109 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:06.114101 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:25:06.114148 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:06.114133 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:25:07.118906 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:07.118875 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" event={"ID":"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc","Type":"ContainerStarted","Data":"d965dd94ae4706c76a3640c8666f474d11d2a303bfadfd0b2e0ee7cfb41d67a9"} Sep 14 11:25:08.041315 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:08.041283 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:25:08.041558 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:08.041535 2556 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 14 11:25:08.041615 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:08.041583 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" podUID="11f0f8a3-1879-470f-aedc-5a6dfb04e3bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 14 11:25:09.041675 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:09.041637 2556 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 14 11:25:09.042042 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:09.041694 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" podUID="11f0f8a3-1879-470f-aedc-5a6dfb04e3bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 14 11:25:10.041354 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:10.041316 2556 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 14 11:25:10.041531 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:10.041372 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" podUID="11f0f8a3-1879-470f-aedc-5a6dfb04e3bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 14 11:25:11.041824 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:11.041790 2556 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 14 11:25:11.042207 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:11.041844 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" podUID="11f0f8a3-1879-470f-aedc-5a6dfb04e3bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 14 11:25:12.041795 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:12.041760 2556 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 14 11:25:12.041964 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:12.041812 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" podUID="11f0f8a3-1879-470f-aedc-5a6dfb04e3bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 14 11:25:13.045774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:13.045744 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:25:13.046195 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:13.046028 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:25:13.046740 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:13.046725 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:25:33.774349 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:33.774316 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7cc544b6-qcsjl"] Sep 14 11:25:33.776716 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:33.776699 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7cc544b6-qcsjl" Sep 14 11:25:33.779580 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:33.779553 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f98vt\"" Sep 14 11:25:33.779580 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:33.779580 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 11:25:33.789197 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:33.789175 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7cc544b6-qcsjl"] Sep 14 11:25:33.846534 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:33.846504 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04c0eac8-0cbc-4a24-90c4-2a020ff21084-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7cc544b6-qcsjl\" (UID: \"04c0eac8-0cbc-4a24-90c4-2a020ff21084\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7cc544b6-qcsjl" Sep 14 11:25:33.947014 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:33.946981 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04c0eac8-0cbc-4a24-90c4-2a020ff21084-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7cc544b6-qcsjl\" (UID: \"04c0eac8-0cbc-4a24-90c4-2a020ff21084\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7cc544b6-qcsjl" Sep 14 11:25:33.947615 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:33.947590 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04c0eac8-0cbc-4a24-90c4-2a020ff21084-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7cc544b6-qcsjl\" (UID: \"04c0eac8-0cbc-4a24-90c4-2a020ff21084\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7cc544b6-qcsjl" Sep 14 11:25:34.087540 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:34.087480 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7cc544b6-qcsjl" Sep 14 11:25:34.355901 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:34.355877 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7cc544b6-qcsjl"] Sep 14 11:25:34.358010 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:25:34.357983 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c0eac8_0cbc_4a24_90c4_2a020ff21084.slice/crio-0884f8479ac9015ab11ca2c8ffe89f543761a3f00a81ec3f12c09b897cffd4f8 WatchSource:0}: Error finding container 0884f8479ac9015ab11ca2c8ffe89f543761a3f00a81ec3f12c09b897cffd4f8: Status 404 returned error can't find the container with id 0884f8479ac9015ab11ca2c8ffe89f543761a3f00a81ec3f12c09b897cffd4f8 Sep 14 11:25:35.279578 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:35.279544 2556 generic.go:356] "Generic (PLEG): container finished" podID="04c0eac8-0cbc-4a24-90c4-2a020ff21084" containerID="b8c7a8aa12e9c63c35e5a523f7d88b993edb0d67718ce6db2fbc7ada6f426d75" exitCode=0 Sep 14 11:25:35.279973 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:35.279602 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7cc544b6-qcsjl" event={"ID":"04c0eac8-0cbc-4a24-90c4-2a020ff21084","Type":"ContainerDied","Data":"b8c7a8aa12e9c63c35e5a523f7d88b993edb0d67718ce6db2fbc7ada6f426d75"} Sep 14 11:25:35.279973 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:35.279650 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7cc544b6-qcsjl" event={"ID":"04c0eac8-0cbc-4a24-90c4-2a020ff21084","Type":"ContainerStarted","Data":"0884f8479ac9015ab11ca2c8ffe89f543761a3f00a81ec3f12c09b897cffd4f8"} Sep 14 11:25:36.409577 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:36.409554 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7cc544b6-qcsjl" Sep 14 11:25:36.473521 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:36.473497 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04c0eac8-0cbc-4a24-90c4-2a020ff21084-maas-api-service-ca\") pod \"04c0eac8-0cbc-4a24-90c4-2a020ff21084\" (UID: \"04c0eac8-0cbc-4a24-90c4-2a020ff21084\") " Sep 14 11:25:36.473825 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:36.473803 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04c0eac8-0cbc-4a24-90c4-2a020ff21084-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "04c0eac8-0cbc-4a24-90c4-2a020ff21084" (UID: "04c0eac8-0cbc-4a24-90c4-2a020ff21084"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:25:36.577015 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:36.575166 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04c0eac8-0cbc-4a24-90c4-2a020ff21084-maas-api-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:25:37.291831 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:37.291795 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7cc544b6-qcsjl" Sep 14 11:25:37.291831 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:37.291814 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7cc544b6-qcsjl" event={"ID":"04c0eac8-0cbc-4a24-90c4-2a020ff21084","Type":"ContainerDied","Data":"0884f8479ac9015ab11ca2c8ffe89f543761a3f00a81ec3f12c09b897cffd4f8"} Sep 14 11:25:37.292103 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:37.291857 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0884f8479ac9015ab11ca2c8ffe89f543761a3f00a81ec3f12c09b897cffd4f8" Sep 14 11:25:49.392132 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:49.392076 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj"] Sep 14 11:25:49.392575 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:49.392343 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj" podUID="6327e7cc-7014-4b42-9207-1ff80b041bca" containerName="payload-processing" containerID="cri-o://50544dda66b2a408d9471f1d0d30fb082fdc7c284494b48636f66c5c2efada9a" gracePeriod=30 Sep 14 11:25:49.639928 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:49.639834 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7cc544b6-qcsjl"] Sep 14 11:25:49.643224 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:49.643148 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7cc544b6-qcsjl"] Sep 14 11:25:49.667934 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:49.667912 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj" Sep 14 11:25:49.815957 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:49.815927 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6327e7cc-7014-4b42-9207-1ff80b041bca-plugins-config-volume\") pod \"6327e7cc-7014-4b42-9207-1ff80b041bca\" (UID: \"6327e7cc-7014-4b42-9207-1ff80b041bca\") " Sep 14 11:25:49.816145 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:49.816026 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9mjl\" (UniqueName: \"kubernetes.io/projected/6327e7cc-7014-4b42-9207-1ff80b041bca-kube-api-access-n9mjl\") pod \"6327e7cc-7014-4b42-9207-1ff80b041bca\" (UID: \"6327e7cc-7014-4b42-9207-1ff80b041bca\") " Sep 14 11:25:49.818148 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:49.816518 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6327e7cc-7014-4b42-9207-1ff80b041bca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6327e7cc-7014-4b42-9207-1ff80b041bca" (UID: "6327e7cc-7014-4b42-9207-1ff80b041bca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:25:49.822896 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:49.818903 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6327e7cc-7014-4b42-9207-1ff80b041bca-kube-api-access-n9mjl" (OuterVolumeSpecName: "kube-api-access-n9mjl") pod "6327e7cc-7014-4b42-9207-1ff80b041bca" (UID: "6327e7cc-7014-4b42-9207-1ff80b041bca"). InnerVolumeSpecName "kube-api-access-n9mjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:25:49.917917 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:49.917819 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9mjl\" (UniqueName: \"kubernetes.io/projected/6327e7cc-7014-4b42-9207-1ff80b041bca-kube-api-access-n9mjl\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:25:49.917917 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:49.917854 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6327e7cc-7014-4b42-9207-1ff80b041bca-plugins-config-volume\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:25:50.360445 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:50.360406 2556 generic.go:356] "Generic (PLEG): container finished" podID="6327e7cc-7014-4b42-9207-1ff80b041bca" containerID="50544dda66b2a408d9471f1d0d30fb082fdc7c284494b48636f66c5c2efada9a" exitCode=0 Sep 14 11:25:50.360621 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:50.360471 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj" Sep 14 11:25:50.360621 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:50.360485 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj" event={"ID":"6327e7cc-7014-4b42-9207-1ff80b041bca","Type":"ContainerDied","Data":"50544dda66b2a408d9471f1d0d30fb082fdc7c284494b48636f66c5c2efada9a"} Sep 14 11:25:50.360621 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:50.360561 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj" event={"ID":"6327e7cc-7014-4b42-9207-1ff80b041bca","Type":"ContainerDied","Data":"262cb49ec899d172b67016d6502e29bbf8843392533b8907b346fb7a40b3cc7f"} Sep 14 11:25:50.360621 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:50.360582 2556 scope.go:117] "RemoveContainer" containerID="50544dda66b2a408d9471f1d0d30fb082fdc7c284494b48636f66c5c2efada9a" Sep 14 11:25:50.369792 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:50.369778 2556 scope.go:117] "RemoveContainer" containerID="50544dda66b2a408d9471f1d0d30fb082fdc7c284494b48636f66c5c2efada9a" Sep 14 11:25:50.370046 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:25:50.370028 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50544dda66b2a408d9471f1d0d30fb082fdc7c284494b48636f66c5c2efada9a\": container with ID starting with 50544dda66b2a408d9471f1d0d30fb082fdc7c284494b48636f66c5c2efada9a not found: ID does not exist" containerID="50544dda66b2a408d9471f1d0d30fb082fdc7c284494b48636f66c5c2efada9a" Sep 14 11:25:50.370135 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:50.370058 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50544dda66b2a408d9471f1d0d30fb082fdc7c284494b48636f66c5c2efada9a"} err="failed to get container status \"50544dda66b2a408d9471f1d0d30fb082fdc7c284494b48636f66c5c2efada9a\": rpc error: code = NotFound desc = could not find container \"50544dda66b2a408d9471f1d0d30fb082fdc7c284494b48636f66c5c2efada9a\": container with ID starting with 50544dda66b2a408d9471f1d0d30fb082fdc7c284494b48636f66c5c2efada9a not found: ID does not exist" Sep 14 11:25:50.381683 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:50.381661 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04c0eac8-0cbc-4a24-90c4-2a020ff21084" path="/var/lib/kubelet/pods/04c0eac8-0cbc-4a24-90c4-2a020ff21084/volumes" Sep 14 11:25:50.388260 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:50.388237 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj"] Sep 14 11:25:50.390848 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:50.390827 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7cc544b6-d59c47f85-tz9pj"] Sep 14 11:25:52.382676 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:25:52.382636 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6327e7cc-7014-4b42-9207-1ff80b041bca" path="/var/lib/kubelet/pods/6327e7cc-7014-4b42-9207-1ff80b041bca/volumes" Sep 14 11:26:30.454070 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.454039 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g"] Sep 14 11:26:30.454570 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.454420 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6327e7cc-7014-4b42-9207-1ff80b041bca" containerName="payload-processing" Sep 14 11:26:30.454570 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.454439 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6327e7cc-7014-4b42-9207-1ff80b041bca" containerName="payload-processing" Sep 14 11:26:30.454570 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.454464 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04c0eac8-0cbc-4a24-90c4-2a020ff21084" containerName="revoke-keys" Sep 14 11:26:30.454570 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.454474 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="04c0eac8-0cbc-4a24-90c4-2a020ff21084" containerName="revoke-keys" Sep 14 11:26:30.454715 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.454575 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6327e7cc-7014-4b42-9207-1ff80b041bca" containerName="payload-processing" Sep 14 11:26:30.454715 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.454588 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="04c0eac8-0cbc-4a24-90c4-2a020ff21084" containerName="revoke-keys" Sep 14 11:26:30.457484 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.457470 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:26:30.471557 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.471531 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-de0f7a\"/\"distinct-2-w4-de0f7a-kserve-self-signed-certs\"" Sep 14 11:26:30.471557 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.471544 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-de0f7a\"/\"kube-root-ca.crt\"" Sep 14 11:26:30.471796 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.471772 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-de0f7a\"/\"openshift-service-ca.crt\"" Sep 14 11:26:30.471925 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.471910 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-de0f7a\"/\"default-dockercfg-556jc\"" Sep 14 11:26:30.474561 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.474536 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g"] Sep 14 11:26:30.571973 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.571942 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75ff5d71-327a-4204-973a-2e561b2cf2dc-home\") pod \"distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g\" (UID: \"75ff5d71-327a-4204-973a-2e561b2cf2dc\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:26:30.572132 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.571984 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ff5d71-327a-4204-973a-2e561b2cf2dc-tmp-dir\") pod \"distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g\" (UID: \"75ff5d71-327a-4204-973a-2e561b2cf2dc\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:26:30.572132 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.572028 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ff5d71-327a-4204-973a-2e561b2cf2dc-model-cache\") pod \"distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g\" (UID: \"75ff5d71-327a-4204-973a-2e561b2cf2dc\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:26:30.572132 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.572097 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps88d\" (UniqueName: \"kubernetes.io/projected/75ff5d71-327a-4204-973a-2e561b2cf2dc-kube-api-access-ps88d\") pod \"distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g\" (UID: \"75ff5d71-327a-4204-973a-2e561b2cf2dc\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:26:30.572253 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.572140 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ff5d71-327a-4204-973a-2e561b2cf2dc-dshm\") pod \"distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g\" (UID: \"75ff5d71-327a-4204-973a-2e561b2cf2dc\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:26:30.572253 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.572165 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ff5d71-327a-4204-973a-2e561b2cf2dc-tls-certs\") pod \"distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g\" (UID: \"75ff5d71-327a-4204-973a-2e561b2cf2dc\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:26:30.672904 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.672863 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ff5d71-327a-4204-973a-2e561b2cf2dc-tmp-dir\") pod \"distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g\" (UID: \"75ff5d71-327a-4204-973a-2e561b2cf2dc\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:26:30.673128 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.672919 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ff5d71-327a-4204-973a-2e561b2cf2dc-model-cache\") pod \"distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g\" (UID: \"75ff5d71-327a-4204-973a-2e561b2cf2dc\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:26:30.673128 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.672990 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps88d\" (UniqueName: \"kubernetes.io/projected/75ff5d71-327a-4204-973a-2e561b2cf2dc-kube-api-access-ps88d\") pod \"distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g\" (UID: \"75ff5d71-327a-4204-973a-2e561b2cf2dc\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:26:30.673128 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.673024 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ff5d71-327a-4204-973a-2e561b2cf2dc-dshm\") pod \"distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g\" (UID: \"75ff5d71-327a-4204-973a-2e561b2cf2dc\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:26:30.673128 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.673058 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ff5d71-327a-4204-973a-2e561b2cf2dc-tls-certs\") pod \"distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g\" (UID: \"75ff5d71-327a-4204-973a-2e561b2cf2dc\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:26:30.673373 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.673157 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75ff5d71-327a-4204-973a-2e561b2cf2dc-home\") pod \"distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g\" (UID: \"75ff5d71-327a-4204-973a-2e561b2cf2dc\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:26:30.673373 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.673298 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ff5d71-327a-4204-973a-2e561b2cf2dc-tmp-dir\") pod \"distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g\" (UID: \"75ff5d71-327a-4204-973a-2e561b2cf2dc\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:26:30.673373 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.673344 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ff5d71-327a-4204-973a-2e561b2cf2dc-model-cache\") pod \"distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g\" (UID: \"75ff5d71-327a-4204-973a-2e561b2cf2dc\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:26:30.673542 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.673526 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75ff5d71-327a-4204-973a-2e561b2cf2dc-home\") pod \"distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g\" (UID: \"75ff5d71-327a-4204-973a-2e561b2cf2dc\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:26:30.675202 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.675180 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ff5d71-327a-4204-973a-2e561b2cf2dc-dshm\") pod \"distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g\" (UID: \"75ff5d71-327a-4204-973a-2e561b2cf2dc\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:26:30.675559 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.675538 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ff5d71-327a-4204-973a-2e561b2cf2dc-tls-certs\") pod \"distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g\" (UID: \"75ff5d71-327a-4204-973a-2e561b2cf2dc\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:26:30.682023 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.682000 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps88d\" (UniqueName: \"kubernetes.io/projected/75ff5d71-327a-4204-973a-2e561b2cf2dc-kube-api-access-ps88d\") pod \"distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g\" (UID: \"75ff5d71-327a-4204-973a-2e561b2cf2dc\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:26:30.765748 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:30.765694 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:26:31.048179 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:31.048145 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g"] Sep 14 11:26:31.048658 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:26:31.048635 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75ff5d71_327a_4204_973a_2e561b2cf2dc.slice/crio-a8d1591894065db9419ddd45cd856c2496539c845fffecd3784e280e83d5be6b WatchSource:0}: Error finding container a8d1591894065db9419ddd45cd856c2496539c845fffecd3784e280e83d5be6b: Status 404 returned error can't find the container with id a8d1591894065db9419ddd45cd856c2496539c845fffecd3784e280e83d5be6b Sep 14 11:26:31.595360 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:31.595325 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" event={"ID":"75ff5d71-327a-4204-973a-2e561b2cf2dc","Type":"ContainerStarted","Data":"c0101badae11d15f2186e751b727c4607df0c655a1e01a4cbd885178be3411ca"} Sep 14 11:26:31.595360 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:31.595364 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" event={"ID":"75ff5d71-327a-4204-973a-2e561b2cf2dc","Type":"ContainerStarted","Data":"a8d1591894065db9419ddd45cd856c2496539c845fffecd3784e280e83d5be6b"} Sep 14 11:26:31.618426 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:31.618382 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" podStartSLOduration=1.618369247 podStartE2EDuration="1.618369247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:26:31.617465764 +0000 UTC m=+1549.680868673" watchObservedRunningTime="2026-09-14 11:26:31.618369247 +0000 UTC m=+1549.681772157" Sep 14 11:26:40.765890 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:40.765859 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:26:40.766259 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:40.765929 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:26:40.778145 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:40.778126 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:26:41.663698 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:41.663670 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:26:56.806030 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:56.805995 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr"] Sep 14 11:26:56.806628 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:56.806543 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" podUID="ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6" containerName="istio-proxy" containerID="cri-o://a323e8d538a2ebea144f7b1b1fa563152a3f83cbd33fb2c1c4ae6becef53423c" gracePeriod=30 Sep 14 11:26:59.586817 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:59.586780 2556 patch_prober.go:28] interesting pod/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 14 11:26:59.587448 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:26:59.587392 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" podUID="ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 14 11:27:02.049308 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.049281 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:27:02.168779 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.168751 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istiod-ca-cert\") pod \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " Sep 14 11:27:02.168942 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.168807 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-workload-socket\") pod \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " Sep 14 11:27:02.168942 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.168841 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qt8n9\" (UniqueName: \"kubernetes.io/projected/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-kube-api-access-qt8n9\") pod \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " Sep 14 11:27:02.168942 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.168894 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istio-token\") pod \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " Sep 14 11:27:02.168942 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.168928 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-workload-certs\") pod \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " Sep 14 11:27:02.169194 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.168950 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istio-podinfo\") pod \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " Sep 14 11:27:02.169194 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.168981 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istio-envoy\") pod \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " Sep 14 11:27:02.169194 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.169006 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-credential-socket\") pod \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " Sep 14 11:27:02.169194 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.169044 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istio-data\") pod \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\" (UID: \"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6\") " Sep 14 11:27:02.169194 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.169162 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6" (UID: "ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:27:02.169453 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.169207 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6" (UID: "ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:27:02.169453 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.169212 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6" (UID: "ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:27:02.169453 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.169369 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istio-data" (OuterVolumeSpecName: "istio-data") pod "ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6" (UID: "ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:27:02.169581 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.169464 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6" (UID: "ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:27:02.169581 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.169485 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:27:02.169581 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.169502 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:27:02.169581 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.169517 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:27:02.169581 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.169529 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:27:02.171004 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.170974 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-kube-api-access-qt8n9" (OuterVolumeSpecName: "kube-api-access-qt8n9") pod "ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6" (UID: "ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6"). InnerVolumeSpecName "kube-api-access-qt8n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:27:02.171161 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.171037 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istio-token" (OuterVolumeSpecName: "istio-token") pod "ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6" (UID: "ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:27:02.171408 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.171378 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6" (UID: "ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:27:02.171408 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.171380 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6" (UID: "ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 11:27:02.270560 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.270527 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt8n9\" (UniqueName: \"kubernetes.io/projected/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-kube-api-access-qt8n9\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:27:02.270560 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.270554 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:27:02.270730 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.270566 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:27:02.270730 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.270577 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:27:02.270730 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.270588 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:27:02.766423 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.766391 2556 generic.go:356] "Generic (PLEG): container finished" podID="ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6" containerID="a323e8d538a2ebea144f7b1b1fa563152a3f83cbd33fb2c1c4ae6becef53423c" exitCode=0 Sep 14 11:27:02.766579 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.766441 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" Sep 14 11:27:02.766579 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.766445 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" event={"ID":"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6","Type":"ContainerDied","Data":"a323e8d538a2ebea144f7b1b1fa563152a3f83cbd33fb2c1c4ae6becef53423c"} Sep 14 11:27:02.766579 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.766503 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr" event={"ID":"ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6","Type":"ContainerDied","Data":"00b57b98b770dd32fa6f5b89cd99747955176150d7a386acc4e44c19bf80d1e9"} Sep 14 11:27:02.766579 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.766533 2556 scope.go:117] "RemoveContainer" containerID="a323e8d538a2ebea144f7b1b1fa563152a3f83cbd33fb2c1c4ae6becef53423c" Sep 14 11:27:02.775059 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.775041 2556 scope.go:117] "RemoveContainer" containerID="de5c3cf8e282799d90e34a35bcfb42e32c9a68ecd903bf8af3ad3935c6b09f2c" Sep 14 11:27:02.783083 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.783052 2556 scope.go:117] "RemoveContainer" containerID="a323e8d538a2ebea144f7b1b1fa563152a3f83cbd33fb2c1c4ae6becef53423c" Sep 14 11:27:02.783393 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:27:02.783369 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a323e8d538a2ebea144f7b1b1fa563152a3f83cbd33fb2c1c4ae6becef53423c\": container with ID starting with a323e8d538a2ebea144f7b1b1fa563152a3f83cbd33fb2c1c4ae6becef53423c not found: ID does not exist" containerID="a323e8d538a2ebea144f7b1b1fa563152a3f83cbd33fb2c1c4ae6becef53423c" Sep 14 11:27:02.783504 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.783402 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a323e8d538a2ebea144f7b1b1fa563152a3f83cbd33fb2c1c4ae6becef53423c"} err="failed to get container status \"a323e8d538a2ebea144f7b1b1fa563152a3f83cbd33fb2c1c4ae6becef53423c\": rpc error: code = NotFound desc = could not find container \"a323e8d538a2ebea144f7b1b1fa563152a3f83cbd33fb2c1c4ae6becef53423c\": container with ID starting with a323e8d538a2ebea144f7b1b1fa563152a3f83cbd33fb2c1c4ae6becef53423c not found: ID does not exist" Sep 14 11:27:02.783504 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.783426 2556 scope.go:117] "RemoveContainer" containerID="de5c3cf8e282799d90e34a35bcfb42e32c9a68ecd903bf8af3ad3935c6b09f2c" Sep 14 11:27:02.783754 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:27:02.783723 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5c3cf8e282799d90e34a35bcfb42e32c9a68ecd903bf8af3ad3935c6b09f2c\": container with ID starting with de5c3cf8e282799d90e34a35bcfb42e32c9a68ecd903bf8af3ad3935c6b09f2c not found: ID does not exist" containerID="de5c3cf8e282799d90e34a35bcfb42e32c9a68ecd903bf8af3ad3935c6b09f2c" Sep 14 11:27:02.783810 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.783763 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5c3cf8e282799d90e34a35bcfb42e32c9a68ecd903bf8af3ad3935c6b09f2c"} err="failed to get container status \"de5c3cf8e282799d90e34a35bcfb42e32c9a68ecd903bf8af3ad3935c6b09f2c\": rpc error: code = NotFound desc = could not find container \"de5c3cf8e282799d90e34a35bcfb42e32c9a68ecd903bf8af3ad3935c6b09f2c\": container with ID starting with de5c3cf8e282799d90e34a35bcfb42e32c9a68ecd903bf8af3ad3935c6b09f2c not found: ID does not exist" Sep 14 11:27:02.789602 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.789578 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr"] Sep 14 11:27:02.792661 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:02.792641 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7cc544b6-openshift-default-c6f48bdb4-67xzr"] Sep 14 11:27:04.382761 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:04.382722 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6" path="/var/lib/kubelet/pods/ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6/volumes" Sep 14 11:27:23.332412 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.332381 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng"] Sep 14 11:27:23.332781 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.332726 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6" containerName="istio-proxy" Sep 14 11:27:23.332781 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.332736 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6" containerName="istio-proxy" Sep 14 11:27:23.332781 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.332745 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6" containerName="istio-proxy" Sep 14 11:27:23.332781 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.332750 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6" containerName="istio-proxy" Sep 14 11:27:23.332929 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.332808 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6" containerName="istio-proxy" Sep 14 11:27:23.332929 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.332815 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac1b2d34-18c4-4f8c-80fe-55bd8d5ecef6" containerName="istio-proxy" Sep 14 11:27:23.335645 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.335627 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:27:23.338216 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.338196 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6f3d0f\"/\"embedding-w0-6f3d0f-kserve-self-signed-certs\"" Sep 14 11:27:23.358476 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.358451 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng"] Sep 14 11:27:23.450880 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.450834 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a6f0275-dd16-4161-a0cf-d5ade98729d3-tmp-dir\") pod \"embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng\" (UID: \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\") " pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:27:23.451081 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.450900 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a6f0275-dd16-4161-a0cf-d5ade98729d3-dshm\") pod \"embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng\" (UID: \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\") " pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:27:23.451081 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.450968 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a6f0275-dd16-4161-a0cf-d5ade98729d3-home\") pod \"embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng\" (UID: \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\") " pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:27:23.451081 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.451033 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a6f0275-dd16-4161-a0cf-d5ade98729d3-model-cache\") pod \"embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng\" (UID: \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\") " pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:27:23.451081 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.451055 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a6f0275-dd16-4161-a0cf-d5ade98729d3-tls-certs\") pod \"embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng\" (UID: \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\") " pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:27:23.451319 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.451083 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mscg\" (UniqueName: \"kubernetes.io/projected/3a6f0275-dd16-4161-a0cf-d5ade98729d3-kube-api-access-6mscg\") pod \"embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng\" (UID: \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\") " pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:27:23.552475 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.552445 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a6f0275-dd16-4161-a0cf-d5ade98729d3-home\") pod \"embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng\" (UID: \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\") " pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:27:23.552633 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.552496 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a6f0275-dd16-4161-a0cf-d5ade98729d3-model-cache\") pod \"embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng\" (UID: \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\") " pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:27:23.552633 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.552518 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a6f0275-dd16-4161-a0cf-d5ade98729d3-tls-certs\") pod \"embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng\" (UID: \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\") " pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:27:23.552633 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.552540 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mscg\" (UniqueName: \"kubernetes.io/projected/3a6f0275-dd16-4161-a0cf-d5ade98729d3-kube-api-access-6mscg\") pod \"embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng\" (UID: \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\") " pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:27:23.552633 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.552600 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a6f0275-dd16-4161-a0cf-d5ade98729d3-tmp-dir\") pod \"embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng\" (UID: \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\") " pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:27:23.552633 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.552631 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a6f0275-dd16-4161-a0cf-d5ade98729d3-dshm\") pod \"embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng\" (UID: \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\") " pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:27:23.552923 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.552899 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a6f0275-dd16-4161-a0cf-d5ade98729d3-home\") pod \"embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng\" (UID: \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\") " pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:27:23.552968 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.552934 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a6f0275-dd16-4161-a0cf-d5ade98729d3-model-cache\") pod \"embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng\" (UID: \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\") " pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:27:23.553010 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.552988 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a6f0275-dd16-4161-a0cf-d5ade98729d3-tmp-dir\") pod \"embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng\" (UID: \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\") " pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:27:23.554827 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.554809 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a6f0275-dd16-4161-a0cf-d5ade98729d3-dshm\") pod \"embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng\" (UID: \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\") " pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:27:23.554979 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.554962 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a6f0275-dd16-4161-a0cf-d5ade98729d3-tls-certs\") pod \"embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng\" (UID: \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\") " pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:27:23.561272 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.561256 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mscg\" (UniqueName: \"kubernetes.io/projected/3a6f0275-dd16-4161-a0cf-d5ade98729d3-kube-api-access-6mscg\") pod \"embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng\" (UID: \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\") " pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:27:23.644692 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.644673 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:27:23.916492 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:23.916420 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng"] Sep 14 11:27:23.919429 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:27:23.919405 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a6f0275_dd16_4161_a0cf_d5ade98729d3.slice/crio-9cae18dd628d5f28fdc933fe917d3a74c78188a76895226a9dc7c0abdecef4f7 WatchSource:0}: Error finding container 9cae18dd628d5f28fdc933fe917d3a74c78188a76895226a9dc7c0abdecef4f7: Status 404 returned error can't find the container with id 9cae18dd628d5f28fdc933fe917d3a74c78188a76895226a9dc7c0abdecef4f7 Sep 14 11:27:24.877934 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:24.877898 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" event={"ID":"3a6f0275-dd16-4161-a0cf-d5ade98729d3","Type":"ContainerStarted","Data":"6a9deab21e246b249ca5fad39d03e00c55ad92177368be1978068a80593eabb4"} Sep 14 11:27:24.877934 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:24.877939 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" event={"ID":"3a6f0275-dd16-4161-a0cf-d5ade98729d3","Type":"ContainerStarted","Data":"9cae18dd628d5f28fdc933fe917d3a74c78188a76895226a9dc7c0abdecef4f7"} Sep 14 11:27:24.898944 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:24.898891 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" podStartSLOduration=1.898876649 podStartE2EDuration="1.898876649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:27:24.897794693 +0000 UTC m=+1602.961197620" watchObservedRunningTime="2026-09-14 11:27:24.898876649 +0000 UTC m=+1602.962279558" Sep 14 11:27:33.645446 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:33.645407 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:27:33.646078 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:33.645508 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:27:33.660302 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:33.660280 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:27:33.943535 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:33.943443 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:27:52.126484 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.126449 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq"] Sep 14 11:27:52.129867 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.129845 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.134554 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.134533 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e4fc87-openshift-default-dockercfg-4ckwt\"" Sep 14 11:27:52.144258 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.144235 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq"] Sep 14 11:27:52.206894 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.206867 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istio-data\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.207014 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.206902 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istio-token\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.207014 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.206932 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istio-podinfo\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.207014 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.206995 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ktjt\" (UniqueName: \"kubernetes.io/projected/4359bf25-5e49-4a8d-92b5-5fd66271b51f-kube-api-access-7ktjt\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.207158 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.207118 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istiod-ca-cert\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.207193 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.207181 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-workload-socket\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.207227 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.207203 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-workload-certs\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.207260 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.207224 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-credential-socket\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.207260 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.207252 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istio-envoy\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.308047 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.308005 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istio-podinfo\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.308841 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.308816 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ktjt\" (UniqueName: \"kubernetes.io/projected/4359bf25-5e49-4a8d-92b5-5fd66271b51f-kube-api-access-7ktjt\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.309504 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.309484 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istiod-ca-cert\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.310466 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.310448 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-workload-socket\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.310607 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.310592 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-workload-certs\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.317917 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.317888 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-credential-socket\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.318081 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.318063 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istio-envoy\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.318331 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.318314 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istio-data\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.318443 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.318430 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istio-token\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.319180 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.319161 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-workload-socket\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.319432 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.319409 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-credential-socket\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.320019 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.310362 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istiod-ca-cert\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.321177 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.321134 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istio-data\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.321507 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.321473 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-workload-certs\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.322110 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.322075 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istio-podinfo\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.326144 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.324574 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istio-envoy\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.326825 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.326768 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ktjt\" (UniqueName: \"kubernetes.io/projected/4359bf25-5e49-4a8d-92b5-5fd66271b51f-kube-api-access-7ktjt\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.328991 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.328928 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istio-token\") pod \"e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.439805 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.439747 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:52.719968 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.719889 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq"] Sep 14 11:27:52.722243 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:27:52.722217 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4359bf25_5e49_4a8d_92b5_5fd66271b51f.slice/crio-260c19e6b93c440ce38d004d108d5dabb694aeef25cd77068318c06953bd4a4f WatchSource:0}: Error finding container 260c19e6b93c440ce38d004d108d5dabb694aeef25cd77068318c06953bd4a4f: Status 404 returned error can't find the container with id 260c19e6b93c440ce38d004d108d5dabb694aeef25cd77068318c06953bd4a4f Sep 14 11:27:52.724400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.724367 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:27:52.724497 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.724431 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:27:52.724497 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:52.724459 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:27:53.023078 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:53.022986 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" event={"ID":"4359bf25-5e49-4a8d-92b5-5fd66271b51f","Type":"ContainerStarted","Data":"fe1ceec7f0acf65795476ac998cda9e1aeb32980cc1104de26532cf625e6d6ad"} Sep 14 11:27:53.023278 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:53.023115 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" event={"ID":"4359bf25-5e49-4a8d-92b5-5fd66271b51f","Type":"ContainerStarted","Data":"260c19e6b93c440ce38d004d108d5dabb694aeef25cd77068318c06953bd4a4f"} Sep 14 11:27:53.048025 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:53.047971 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" podStartSLOduration=1.047955472 podStartE2EDuration="1.047955472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:27:53.046113362 +0000 UTC m=+1631.109516273" watchObservedRunningTime="2026-09-14 11:27:53.047955472 +0000 UTC m=+1631.111358382" Sep 14 11:27:53.440419 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:53.440384 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:53.445378 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:53.445355 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:54.028066 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:54.028023 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:27:54.029143 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:27:54.029124 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:28:01.260474 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:01.260436 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl"] Sep 14 11:28:01.264258 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:01.264232 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl" Sep 14 11:28:01.266717 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:01.266698 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e4fc87-dockercfg-rlkbf\"" Sep 14 11:28:01.267030 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:01.267013 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e4fc87\"" Sep 14 11:28:01.274721 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:01.274704 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl"] Sep 14 11:28:01.412817 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:01.412781 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1013ed-ced5-49c7-b743-8f4c587222a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl\" (UID: \"ef1013ed-ced5-49c7-b743-8f4c587222a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl" Sep 14 11:28:01.413029 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:01.412894 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwjbw\" (UniqueName: \"kubernetes.io/projected/ef1013ed-ced5-49c7-b743-8f4c587222a6-kube-api-access-lwjbw\") pod \"payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl\" (UID: \"ef1013ed-ced5-49c7-b743-8f4c587222a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl" Sep 14 11:28:01.514332 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:01.514248 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1013ed-ced5-49c7-b743-8f4c587222a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl\" (UID: \"ef1013ed-ced5-49c7-b743-8f4c587222a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl" Sep 14 11:28:01.514511 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:01.514348 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwjbw\" (UniqueName: \"kubernetes.io/projected/ef1013ed-ced5-49c7-b743-8f4c587222a6-kube-api-access-lwjbw\") pod \"payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl\" (UID: \"ef1013ed-ced5-49c7-b743-8f4c587222a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl" Sep 14 11:28:01.514951 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:01.514932 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1013ed-ced5-49c7-b743-8f4c587222a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl\" (UID: \"ef1013ed-ced5-49c7-b743-8f4c587222a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl" Sep 14 11:28:01.527407 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:01.527388 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwjbw\" (UniqueName: \"kubernetes.io/projected/ef1013ed-ced5-49c7-b743-8f4c587222a6-kube-api-access-lwjbw\") pod \"payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl\" (UID: \"ef1013ed-ced5-49c7-b743-8f4c587222a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl" Sep 14 11:28:01.575486 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:01.575461 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl" Sep 14 11:28:01.847432 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:01.847410 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl"] Sep 14 11:28:01.849037 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:28:01.849009 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1013ed_ced5_49c7_b743_8f4c587222a6.slice/crio-486f15518f2328a62befee7112244221d590d2b90ab128d142b3077ac54d13ad WatchSource:0}: Error finding container 486f15518f2328a62befee7112244221d590d2b90ab128d142b3077ac54d13ad: Status 404 returned error can't find the container with id 486f15518f2328a62befee7112244221d590d2b90ab128d142b3077ac54d13ad Sep 14 11:28:02.067764 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:02.067726 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl" event={"ID":"ef1013ed-ced5-49c7-b743-8f4c587222a6","Type":"ContainerStarted","Data":"8a5d11fa1a47ae1ab6371fd824bc6bea88e6586d5c724f15bec75e8f089c26c2"} Sep 14 11:28:02.067921 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:02.067788 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl" event={"ID":"ef1013ed-ced5-49c7-b743-8f4c587222a6","Type":"ContainerStarted","Data":"486f15518f2328a62befee7112244221d590d2b90ab128d142b3077ac54d13ad"} Sep 14 11:28:02.067921 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:02.067901 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl" Sep 14 11:28:02.088344 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:02.088288 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl" podStartSLOduration=1.088270592 podStartE2EDuration="1.088270592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:28:02.086331995 +0000 UTC m=+1640.149734906" watchObservedRunningTime="2026-09-14 11:28:02.088270592 +0000 UTC m=+1640.151673503" Sep 14 11:28:13.074372 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:13.074340 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl" Sep 14 11:28:19.155832 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:19.155802 2556 generic.go:356] "Generic (PLEG): container finished" podID="4359bf25-5e49-4a8d-92b5-5fd66271b51f" containerID="fe1ceec7f0acf65795476ac998cda9e1aeb32980cc1104de26532cf625e6d6ad" exitCode=0 Sep 14 11:28:19.156224 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:19.155877 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" event={"ID":"4359bf25-5e49-4a8d-92b5-5fd66271b51f","Type":"ContainerDied","Data":"fe1ceec7f0acf65795476ac998cda9e1aeb32980cc1104de26532cf625e6d6ad"} Sep 14 11:28:19.156312 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:19.156299 2556 scope.go:117] "RemoveContainer" containerID="fe1ceec7f0acf65795476ac998cda9e1aeb32980cc1104de26532cf625e6d6ad" Sep 14 11:28:19.157481 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:19.157453 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:28:19.157544 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:19.157520 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:28:19.157584 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:19.157549 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:28:20.164569 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:20.164532 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" event={"ID":"4359bf25-5e49-4a8d-92b5-5fd66271b51f","Type":"ContainerStarted","Data":"27d67f26e3976c40a771838b2ce00b6f4a162cdba9c5fea1093324f89a9c057e"} Sep 14 11:28:20.440470 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:20.440374 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:28:20.441604 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:20.441576 2556 patch_prober.go:28] interesting pod/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 14 11:28:20.441710 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:20.441636 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" podUID="4359bf25-5e49-4a8d-92b5-5fd66271b51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 14 11:28:21.440860 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:21.440824 2556 patch_prober.go:28] interesting pod/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 14 11:28:21.441278 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:21.440880 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" podUID="4359bf25-5e49-4a8d-92b5-5fd66271b51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 14 11:28:22.440480 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:22.440446 2556 patch_prober.go:28] interesting pod/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 14 11:28:22.440655 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:22.440498 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" podUID="4359bf25-5e49-4a8d-92b5-5fd66271b51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 14 11:28:23.440145 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:23.440108 2556 patch_prober.go:28] interesting pod/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 14 11:28:23.440527 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:23.440164 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" podUID="4359bf25-5e49-4a8d-92b5-5fd66271b51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 14 11:28:24.028852 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:24.028820 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:28:24.440120 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:24.440070 2556 patch_prober.go:28] interesting pod/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 14 11:28:24.440286 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:24.440159 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" podUID="4359bf25-5e49-4a8d-92b5-5fd66271b51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 14 11:28:25.444820 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.444794 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:28:25.445861 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.445841 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:28:25.758604 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.758532 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst"] Sep 14 11:28:25.761340 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.761322 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.764164 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.764143 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-420d64-openshift-default-dockercfg-vqhpb\"" Sep 14 11:28:25.773317 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.773297 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst"] Sep 14 11:28:25.849980 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.849955 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istio-podinfo\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.850253 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.850229 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-credential-socket\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.850397 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.850372 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istiod-ca-cert\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.850555 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.850536 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istio-envoy\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.850711 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.850693 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq5h2\" (UniqueName: \"kubernetes.io/projected/d7d65916-67c9-42b7-8b55-3311ed30ab2d-kube-api-access-cq5h2\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.850852 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.850836 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istio-data\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.850970 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.850956 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-workload-socket\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.851132 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.851112 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-workload-certs\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.851354 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.851335 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istio-token\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.953009 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.952973 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istio-token\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.953220 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.953025 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istio-podinfo\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.953220 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.953057 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-credential-socket\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.953220 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.953074 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istiod-ca-cert\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.953220 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.953128 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istio-envoy\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.953220 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.953170 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq5h2\" (UniqueName: \"kubernetes.io/projected/d7d65916-67c9-42b7-8b55-3311ed30ab2d-kube-api-access-cq5h2\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.953220 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.953195 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istio-data\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.953220 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.953214 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-workload-socket\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.953598 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.953236 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-workload-certs\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.953598 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.953541 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-credential-socket\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.953709 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.953617 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istio-data\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.953709 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.953651 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-workload-socket\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.953709 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.953689 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-workload-certs\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.953860 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.953841 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istiod-ca-cert\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.955497 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.955479 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istio-envoy\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.955629 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.955611 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istio-podinfo\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.961035 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.961008 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istio-token\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:25.961209 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:25.961192 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq5h2\" (UniqueName: \"kubernetes.io/projected/d7d65916-67c9-42b7-8b55-3311ed30ab2d-kube-api-access-cq5h2\") pod \"e2e-api-b-420d64-openshift-default-5654854486-b5lst\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:26.071844 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:26.071765 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:26.346639 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:26.346615 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst"] Sep 14 11:28:26.349552 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:28:26.349526 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d65916_67c9_42b7_8b55_3311ed30ab2d.slice/crio-c66b77fadd7e31b720ab0c2bad5414f3cad13da5119a8c9f3de42e24911d5b88 WatchSource:0}: Error finding container c66b77fadd7e31b720ab0c2bad5414f3cad13da5119a8c9f3de42e24911d5b88: Status 404 returned error can't find the container with id c66b77fadd7e31b720ab0c2bad5414f3cad13da5119a8c9f3de42e24911d5b88 Sep 14 11:28:26.351839 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:26.351806 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:28:26.351938 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:26.351884 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:28:26.351938 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:26.351929 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:28:27.207767 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:27.207730 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" event={"ID":"d7d65916-67c9-42b7-8b55-3311ed30ab2d","Type":"ContainerStarted","Data":"23abcf0b22754cac9012819f0f6a038e73333c6d383f3c8a841f3ca357ab4933"} Sep 14 11:28:27.208200 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:27.207782 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" event={"ID":"d7d65916-67c9-42b7-8b55-3311ed30ab2d","Type":"ContainerStarted","Data":"c66b77fadd7e31b720ab0c2bad5414f3cad13da5119a8c9f3de42e24911d5b88"} Sep 14 11:28:27.247995 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:27.247946 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" podStartSLOduration=2.24793067 podStartE2EDuration="2.24793067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:28:27.24549256 +0000 UTC m=+1665.308895469" watchObservedRunningTime="2026-09-14 11:28:27.24793067 +0000 UTC m=+1665.311333579" Sep 14 11:28:28.072247 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:28.072212 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:28.076877 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:28.076853 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:28.213373 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:28.213336 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:28.214181 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:28.214159 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:28:35.623971 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:35.623938 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg"] Sep 14 11:28:35.626625 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:35.626608 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg" Sep 14 11:28:35.629289 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:35.629267 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-420d64\"" Sep 14 11:28:35.629421 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:35.629328 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-420d64-dockercfg-9hwp9\"" Sep 14 11:28:35.637604 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:35.637583 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg"] Sep 14 11:28:35.753421 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:35.753383 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mllj\" (UniqueName: \"kubernetes.io/projected/e0e612e7-2a2b-4d68-9e0c-d9b0151e4984-kube-api-access-6mllj\") pod \"payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg\" (UID: \"e0e612e7-2a2b-4d68-9e0c-d9b0151e4984\") " pod="openshift-ingress/payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg" Sep 14 11:28:35.753612 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:35.753459 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e612e7-2a2b-4d68-9e0c-d9b0151e4984-plugins-config-volume\") pod \"payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg\" (UID: \"e0e612e7-2a2b-4d68-9e0c-d9b0151e4984\") " pod="openshift-ingress/payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg" Sep 14 11:28:35.854145 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:35.854113 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mllj\" (UniqueName: \"kubernetes.io/projected/e0e612e7-2a2b-4d68-9e0c-d9b0151e4984-kube-api-access-6mllj\") pod \"payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg\" (UID: \"e0e612e7-2a2b-4d68-9e0c-d9b0151e4984\") " pod="openshift-ingress/payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg" Sep 14 11:28:35.854322 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:35.854182 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e612e7-2a2b-4d68-9e0c-d9b0151e4984-plugins-config-volume\") pod \"payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg\" (UID: \"e0e612e7-2a2b-4d68-9e0c-d9b0151e4984\") " pod="openshift-ingress/payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg" Sep 14 11:28:35.854841 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:35.854821 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e612e7-2a2b-4d68-9e0c-d9b0151e4984-plugins-config-volume\") pod \"payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg\" (UID: \"e0e612e7-2a2b-4d68-9e0c-d9b0151e4984\") " pod="openshift-ingress/payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg" Sep 14 11:28:35.862437 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:35.862411 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mllj\" (UniqueName: \"kubernetes.io/projected/e0e612e7-2a2b-4d68-9e0c-d9b0151e4984-kube-api-access-6mllj\") pod \"payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg\" (UID: \"e0e612e7-2a2b-4d68-9e0c-d9b0151e4984\") " pod="openshift-ingress/payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg" Sep 14 11:28:35.937608 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:35.937544 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg" Sep 14 11:28:36.208667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:36.208616 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg"] Sep 14 11:28:36.211079 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:28:36.211051 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e612e7_2a2b_4d68_9e0c_d9b0151e4984.slice/crio-c661a272d8cc93bc6d154d5c225f15538b85baa0d8c5c6fc2ba7ff3014077487 WatchSource:0}: Error finding container c661a272d8cc93bc6d154d5c225f15538b85baa0d8c5c6fc2ba7ff3014077487: Status 404 returned error can't find the container with id c661a272d8cc93bc6d154d5c225f15538b85baa0d8c5c6fc2ba7ff3014077487 Sep 14 11:28:36.261670 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:36.261638 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg" event={"ID":"e0e612e7-2a2b-4d68-9e0c-d9b0151e4984","Type":"ContainerStarted","Data":"c661a272d8cc93bc6d154d5c225f15538b85baa0d8c5c6fc2ba7ff3014077487"} Sep 14 11:28:37.268213 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:37.268169 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg" event={"ID":"e0e612e7-2a2b-4d68-9e0c-d9b0151e4984","Type":"ContainerStarted","Data":"4e24e24c710c287cbeeb6fde8fadd123d8e0204ad884eafca8fc103b8547a290"} Sep 14 11:28:37.268668 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:37.268237 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg" Sep 14 11:28:37.288642 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:37.288594 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg" podStartSLOduration=2.288580242 podStartE2EDuration="2.288580242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:28:37.285560296 +0000 UTC m=+1675.348963199" watchObservedRunningTime="2026-09-14 11:28:37.288580242 +0000 UTC m=+1675.351983150" Sep 14 11:28:45.758173 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:45.758138 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6f3d0f-jn7j9"] Sep 14 11:28:45.760610 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:45.760595 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6f3d0f-jn7j9" Sep 14 11:28:45.763227 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:45.763208 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 11:28:45.763351 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:45.763214 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f98vt\"" Sep 14 11:28:45.771121 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:45.771098 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6f3d0f-jn7j9"] Sep 14 11:28:45.857186 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:45.857147 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4576f9-005a-443a-b7d6-f7609eaa60e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6f3d0f-jn7j9\" (UID: \"1a4576f9-005a-443a-b7d6-f7609eaa60e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6f3d0f-jn7j9" Sep 14 11:28:45.958488 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:45.958457 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4576f9-005a-443a-b7d6-f7609eaa60e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6f3d0f-jn7j9\" (UID: \"1a4576f9-005a-443a-b7d6-f7609eaa60e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6f3d0f-jn7j9" Sep 14 11:28:45.959053 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:45.959035 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4576f9-005a-443a-b7d6-f7609eaa60e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6f3d0f-jn7j9\" (UID: \"1a4576f9-005a-443a-b7d6-f7609eaa60e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6f3d0f-jn7j9" Sep 14 11:28:46.071139 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:46.071042 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6f3d0f-jn7j9" Sep 14 11:28:46.357065 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:28:46.357038 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a4576f9_005a_443a_b7d6_f7609eaa60e9.slice/crio-9ef96c44a434ff0770c1db70f46689adb3e9b4991a7ec23739e869679ccbd287 WatchSource:0}: Error finding container 9ef96c44a434ff0770c1db70f46689adb3e9b4991a7ec23739e869679ccbd287: Status 404 returned error can't find the container with id 9ef96c44a434ff0770c1db70f46689adb3e9b4991a7ec23739e869679ccbd287 Sep 14 11:28:46.358625 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:46.358600 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6f3d0f-jn7j9"] Sep 14 11:28:47.324351 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:47.324316 2556 generic.go:356] "Generic (PLEG): container finished" podID="1a4576f9-005a-443a-b7d6-f7609eaa60e9" containerID="48d0f7503ee406280c3b308fa6ce649c7b596ef2981284d91fab94cae4e29bce" exitCode=0 Sep 14 11:28:47.324735 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:47.324377 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6f3d0f-jn7j9" event={"ID":"1a4576f9-005a-443a-b7d6-f7609eaa60e9","Type":"ContainerDied","Data":"48d0f7503ee406280c3b308fa6ce649c7b596ef2981284d91fab94cae4e29bce"} Sep 14 11:28:47.324735 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:47.324455 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6f3d0f-jn7j9" event={"ID":"1a4576f9-005a-443a-b7d6-f7609eaa60e9","Type":"ContainerStarted","Data":"9ef96c44a434ff0770c1db70f46689adb3e9b4991a7ec23739e869679ccbd287"} Sep 14 11:28:48.275947 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:48.275920 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg" Sep 14 11:28:48.459955 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:48.459930 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6f3d0f-jn7j9" Sep 14 11:28:48.486465 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:48.486439 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4576f9-005a-443a-b7d6-f7609eaa60e9-maas-api-service-ca\") pod \"1a4576f9-005a-443a-b7d6-f7609eaa60e9\" (UID: \"1a4576f9-005a-443a-b7d6-f7609eaa60e9\") " Sep 14 11:28:48.486940 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:48.486916 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a4576f9-005a-443a-b7d6-f7609eaa60e9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1a4576f9-005a-443a-b7d6-f7609eaa60e9" (UID: "1a4576f9-005a-443a-b7d6-f7609eaa60e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:28:48.587326 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:48.587261 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4576f9-005a-443a-b7d6-f7609eaa60e9-maas-api-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:28:49.337881 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:49.337841 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6f3d0f-jn7j9" event={"ID":"1a4576f9-005a-443a-b7d6-f7609eaa60e9","Type":"ContainerDied","Data":"9ef96c44a434ff0770c1db70f46689adb3e9b4991a7ec23739e869679ccbd287"} Sep 14 11:28:49.338037 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:49.337888 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ef96c44a434ff0770c1db70f46689adb3e9b4991a7ec23739e869679ccbd287" Sep 14 11:28:49.338037 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:49.337856 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6f3d0f-jn7j9" Sep 14 11:28:50.620903 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:50.620870 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw"] Sep 14 11:28:50.636080 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:50.636052 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng"] Sep 14 11:28:50.647492 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:50.647464 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2"] Sep 14 11:28:56.912901 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:56.912860 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4fc87-jz46g"] Sep 14 11:28:56.913474 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:56.913438 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a4576f9-005a-443a-b7d6-f7609eaa60e9" containerName="revoke-keys" Sep 14 11:28:56.913474 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:56.913459 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a4576f9-005a-443a-b7d6-f7609eaa60e9" containerName="revoke-keys" Sep 14 11:28:56.913598 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:56.913581 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a4576f9-005a-443a-b7d6-f7609eaa60e9" containerName="revoke-keys" Sep 14 11:28:56.915775 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:56.915754 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4fc87-jz46g" Sep 14 11:28:56.919084 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:56.919064 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f98vt\"" Sep 14 11:28:56.919360 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:56.919338 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 11:28:56.932681 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:56.932663 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4fc87-jz46g"] Sep 14 11:28:56.963329 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:56.963299 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa44a509-1e67-4811-b87f-5107e7ca41be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e4fc87-jz46g\" (UID: \"aa44a509-1e67-4811-b87f-5107e7ca41be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4fc87-jz46g" Sep 14 11:28:57.063948 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:57.063925 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa44a509-1e67-4811-b87f-5107e7ca41be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e4fc87-jz46g\" (UID: \"aa44a509-1e67-4811-b87f-5107e7ca41be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4fc87-jz46g" Sep 14 11:28:57.064505 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:57.064488 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa44a509-1e67-4811-b87f-5107e7ca41be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e4fc87-jz46g\" (UID: \"aa44a509-1e67-4811-b87f-5107e7ca41be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4fc87-jz46g" Sep 14 11:28:57.224919 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:57.224858 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4fc87-jz46g" Sep 14 11:28:57.507577 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:57.507554 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4fc87-jz46g"] Sep 14 11:28:57.509377 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:28:57.509352 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa44a509_1e67_4811_b87f_5107e7ca41be.slice/crio-14bc474b184c99e062ce75e9439f1244affab73c61a4f198a7ad30bc06dab960 WatchSource:0}: Error finding container 14bc474b184c99e062ce75e9439f1244affab73c61a4f198a7ad30bc06dab960: Status 404 returned error can't find the container with id 14bc474b184c99e062ce75e9439f1244affab73c61a4f198a7ad30bc06dab960 Sep 14 11:28:58.384561 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:58.384533 2556 generic.go:356] "Generic (PLEG): container finished" podID="aa44a509-1e67-4811-b87f-5107e7ca41be" containerID="efc5b263c3806fb9bd79589b676cd9dad6891c2e6e9b03fc92b6fe4f6e1ee478" exitCode=0 Sep 14 11:28:58.384908 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:58.384627 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4fc87-jz46g" event={"ID":"aa44a509-1e67-4811-b87f-5107e7ca41be","Type":"ContainerDied","Data":"efc5b263c3806fb9bd79589b676cd9dad6891c2e6e9b03fc92b6fe4f6e1ee478"} Sep 14 11:28:58.384908 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:58.384684 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4fc87-jz46g" event={"ID":"aa44a509-1e67-4811-b87f-5107e7ca41be","Type":"ContainerStarted","Data":"14bc474b184c99e062ce75e9439f1244affab73c61a4f198a7ad30bc06dab960"} Sep 14 11:28:59.392647 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:59.392618 2556 generic.go:356] "Generic (PLEG): container finished" podID="32e5fce6-40e2-4456-bc82-d258c4ccf5d1" containerID="097c92e82e2c091f9d4d774626a8d5455ce600c3c207d340342f0075419db0df" exitCode=137 Sep 14 11:28:59.393039 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:59.392687 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65" event={"ID":"32e5fce6-40e2-4456-bc82-d258c4ccf5d1","Type":"ContainerDied","Data":"097c92e82e2c091f9d4d774626a8d5455ce600c3c207d340342f0075419db0df"} Sep 14 11:28:59.393119 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:59.393100 2556 scope.go:117] "RemoveContainer" containerID="097c92e82e2c091f9d4d774626a8d5455ce600c3c207d340342f0075419db0df" Sep 14 11:28:59.546997 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:59.546969 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4fc87-jz46g" Sep 14 11:28:59.589223 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:59.589190 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa44a509-1e67-4811-b87f-5107e7ca41be-maas-api-service-ca\") pod \"aa44a509-1e67-4811-b87f-5107e7ca41be\" (UID: \"aa44a509-1e67-4811-b87f-5107e7ca41be\") " Sep 14 11:28:59.589573 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:59.589546 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa44a509-1e67-4811-b87f-5107e7ca41be-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aa44a509-1e67-4811-b87f-5107e7ca41be" (UID: "aa44a509-1e67-4811-b87f-5107e7ca41be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:28:59.589683 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:28:59.589667 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa44a509-1e67-4811-b87f-5107e7ca41be-maas-api-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:00.399961 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:00.399930 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65" event={"ID":"32e5fce6-40e2-4456-bc82-d258c4ccf5d1","Type":"ContainerStarted","Data":"3928b2e0e8b0e8d992e694ce50d2186981fe89ae9d2aa3690585dbe5c726765f"} Sep 14 11:29:00.400350 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:00.400227 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65" Sep 14 11:29:00.401339 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:00.401319 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4fc87-jz46g" Sep 14 11:29:00.401443 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:00.401351 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4fc87-jz46g" event={"ID":"aa44a509-1e67-4811-b87f-5107e7ca41be","Type":"ContainerDied","Data":"14bc474b184c99e062ce75e9439f1244affab73c61a4f198a7ad30bc06dab960"} Sep 14 11:29:00.401443 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:00.401387 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14bc474b184c99e062ce75e9439f1244affab73c61a4f198a7ad30bc06dab960" Sep 14 11:29:05.583081 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:05.582979 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6f3d0f-jn7j9"] Sep 14 11:29:05.585343 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:05.585316 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6f3d0f-jn7j9"] Sep 14 11:29:05.639260 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:05.639228 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" podUID="1feb9dd5-e808-4402-a88f-6c5726560ac4" containerName="main" containerID="cri-o://01ae59f1fc273123bd58f635aa3a915bb3ae66b653cb59cbea5127704b0e1958" gracePeriod=45 Sep 14 11:29:05.668857 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:05.668826 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" podUID="f4138f69-d4ca-4f25-ae4f-4e889bf8a972" containerName="main" containerID="cri-o://6d17f337dc1c494799da4929a8bf6a7dc95e9cc7c17f9898864c64372ca383f5" gracePeriod=45 Sep 14 11:29:05.834151 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:05.834049 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" podUID="1feb9dd5-e808-4402-a88f-6c5726560ac4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": dial tcp 10.132.0.68:8000: connect: connection refused" Sep 14 11:29:05.955635 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:05.955611 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:29:05.958213 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:05.958191 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:29:06.050151 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.050124 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1feb9dd5-e808-4402-a88f-6c5726560ac4-home\") pod \"1feb9dd5-e808-4402-a88f-6c5726560ac4\" (UID: \"1feb9dd5-e808-4402-a88f-6c5726560ac4\") " Sep 14 11:29:06.050309 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.050161 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1feb9dd5-e808-4402-a88f-6c5726560ac4-dshm\") pod \"1feb9dd5-e808-4402-a88f-6c5726560ac4\" (UID: \"1feb9dd5-e808-4402-a88f-6c5726560ac4\") " Sep 14 11:29:06.050309 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.050202 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-dshm\") pod \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\" (UID: \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\") " Sep 14 11:29:06.050309 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.050236 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1feb9dd5-e808-4402-a88f-6c5726560ac4-tls-certs\") pod \"1feb9dd5-e808-4402-a88f-6c5726560ac4\" (UID: \"1feb9dd5-e808-4402-a88f-6c5726560ac4\") " Sep 14 11:29:06.050309 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.050268 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvx4q\" (UniqueName: \"kubernetes.io/projected/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-kube-api-access-jvx4q\") pod \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\" (UID: \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\") " Sep 14 11:29:06.050309 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.050302 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-home\") pod \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\" (UID: \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\") " Sep 14 11:29:06.050624 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.050335 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-tls-certs\") pod \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\" (UID: \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\") " Sep 14 11:29:06.050624 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.050363 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1feb9dd5-e808-4402-a88f-6c5726560ac4-model-cache\") pod \"1feb9dd5-e808-4402-a88f-6c5726560ac4\" (UID: \"1feb9dd5-e808-4402-a88f-6c5726560ac4\") " Sep 14 11:29:06.050624 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.050443 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-tmp-dir\") pod \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\" (UID: \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\") " Sep 14 11:29:06.050624 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.050442 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1feb9dd5-e808-4402-a88f-6c5726560ac4-home" (OuterVolumeSpecName: "home") pod "1feb9dd5-e808-4402-a88f-6c5726560ac4" (UID: "1feb9dd5-e808-4402-a88f-6c5726560ac4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:06.050624 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.050481 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-model-cache\") pod \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\" (UID: \"f4138f69-d4ca-4f25-ae4f-4e889bf8a972\") " Sep 14 11:29:06.050624 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.050510 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1feb9dd5-e808-4402-a88f-6c5726560ac4-tmp-dir\") pod \"1feb9dd5-e808-4402-a88f-6c5726560ac4\" (UID: \"1feb9dd5-e808-4402-a88f-6c5726560ac4\") " Sep 14 11:29:06.050624 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.050549 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pq4xx\" (UniqueName: \"kubernetes.io/projected/1feb9dd5-e808-4402-a88f-6c5726560ac4-kube-api-access-pq4xx\") pod \"1feb9dd5-e808-4402-a88f-6c5726560ac4\" (UID: \"1feb9dd5-e808-4402-a88f-6c5726560ac4\") " Sep 14 11:29:06.050959 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.050787 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-home" (OuterVolumeSpecName: "home") pod "f4138f69-d4ca-4f25-ae4f-4e889bf8a972" (UID: "f4138f69-d4ca-4f25-ae4f-4e889bf8a972"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:06.050959 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.050805 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4138f69-d4ca-4f25-ae4f-4e889bf8a972" (UID: "f4138f69-d4ca-4f25-ae4f-4e889bf8a972"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:06.051072 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.050975 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1feb9dd5-e808-4402-a88f-6c5726560ac4-model-cache" (OuterVolumeSpecName: "model-cache") pod "1feb9dd5-e808-4402-a88f-6c5726560ac4" (UID: "1feb9dd5-e808-4402-a88f-6c5726560ac4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:06.051072 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.050981 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1feb9dd5-e808-4402-a88f-6c5726560ac4-home\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:06.051072 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.050998 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-home\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:06.051072 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.051022 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-tmp-dir\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:06.051300 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.051173 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4138f69-d4ca-4f25-ae4f-4e889bf8a972" (UID: "f4138f69-d4ca-4f25-ae4f-4e889bf8a972"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:06.051300 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.051220 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1feb9dd5-e808-4402-a88f-6c5726560ac4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1feb9dd5-e808-4402-a88f-6c5726560ac4" (UID: "1feb9dd5-e808-4402-a88f-6c5726560ac4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:06.052851 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.052812 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4138f69-d4ca-4f25-ae4f-4e889bf8a972" (UID: "f4138f69-d4ca-4f25-ae4f-4e889bf8a972"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:29:06.052851 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.052845 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1feb9dd5-e808-4402-a88f-6c5726560ac4-dshm" (OuterVolumeSpecName: "dshm") pod "1feb9dd5-e808-4402-a88f-6c5726560ac4" (UID: "1feb9dd5-e808-4402-a88f-6c5726560ac4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:06.053015 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.052863 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1feb9dd5-e808-4402-a88f-6c5726560ac4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1feb9dd5-e808-4402-a88f-6c5726560ac4" (UID: "1feb9dd5-e808-4402-a88f-6c5726560ac4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:29:06.053054 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.053019 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1feb9dd5-e808-4402-a88f-6c5726560ac4-kube-api-access-pq4xx" (OuterVolumeSpecName: "kube-api-access-pq4xx") pod "1feb9dd5-e808-4402-a88f-6c5726560ac4" (UID: "1feb9dd5-e808-4402-a88f-6c5726560ac4"). InnerVolumeSpecName "kube-api-access-pq4xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:29:06.053286 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.053262 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-dshm" (OuterVolumeSpecName: "dshm") pod "f4138f69-d4ca-4f25-ae4f-4e889bf8a972" (UID: "f4138f69-d4ca-4f25-ae4f-4e889bf8a972"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:06.053286 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.053274 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-kube-api-access-jvx4q" (OuterVolumeSpecName: "kube-api-access-jvx4q") pod "f4138f69-d4ca-4f25-ae4f-4e889bf8a972" (UID: "f4138f69-d4ca-4f25-ae4f-4e889bf8a972"). InnerVolumeSpecName "kube-api-access-jvx4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:29:06.152034 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.152000 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-model-cache\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:06.152034 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.152028 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1feb9dd5-e808-4402-a88f-6c5726560ac4-tmp-dir\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:06.152034 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.152036 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq4xx\" (UniqueName: \"kubernetes.io/projected/1feb9dd5-e808-4402-a88f-6c5726560ac4-kube-api-access-pq4xx\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:06.152276 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.152048 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1feb9dd5-e808-4402-a88f-6c5726560ac4-dshm\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:06.152276 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.152056 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-dshm\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:06.152276 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.152063 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1feb9dd5-e808-4402-a88f-6c5726560ac4-tls-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:06.152276 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.152072 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvx4q\" (UniqueName: \"kubernetes.io/projected/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-kube-api-access-jvx4q\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:06.152276 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.152080 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4138f69-d4ca-4f25-ae4f-4e889bf8a972-tls-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:06.152276 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.152110 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1feb9dd5-e808-4402-a88f-6c5726560ac4-model-cache\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:06.366306 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.366251 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" podUID="3a6f0275-dd16-4161-a0cf-d5ade98729d3" containerName="main" containerID="cri-o://6a9deab21e246b249ca5fad39d03e00c55ad92177368be1978068a80593eabb4" gracePeriod=45 Sep 14 11:29:06.384222 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.384193 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a4576f9-005a-443a-b7d6-f7609eaa60e9" path="/var/lib/kubelet/pods/1a4576f9-005a-443a-b7d6-f7609eaa60e9/volumes" Sep 14 11:29:06.433489 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.433461 2556 generic.go:356] "Generic (PLEG): container finished" podID="f4138f69-d4ca-4f25-ae4f-4e889bf8a972" containerID="6d17f337dc1c494799da4929a8bf6a7dc95e9cc7c17f9898864c64372ca383f5" exitCode=0 Sep 14 11:29:06.433603 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.433508 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" Sep 14 11:29:06.433603 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.433541 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" event={"ID":"f4138f69-d4ca-4f25-ae4f-4e889bf8a972","Type":"ContainerDied","Data":"6d17f337dc1c494799da4929a8bf6a7dc95e9cc7c17f9898864c64372ca383f5"} Sep 14 11:29:06.433603 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.433579 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2" event={"ID":"f4138f69-d4ca-4f25-ae4f-4e889bf8a972","Type":"ContainerDied","Data":"8994dab17ad3639c93bb521a419ff890d6c047908b078b0058c4f3fd530e402c"} Sep 14 11:29:06.433603 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.433596 2556 scope.go:117] "RemoveContainer" containerID="6d17f337dc1c494799da4929a8bf6a7dc95e9cc7c17f9898864c64372ca383f5" Sep 14 11:29:06.435015 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.434985 2556 generic.go:356] "Generic (PLEG): container finished" podID="1feb9dd5-e808-4402-a88f-6c5726560ac4" containerID="01ae59f1fc273123bd58f635aa3a915bb3ae66b653cb59cbea5127704b0e1958" exitCode=0 Sep 14 11:29:06.435143 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.435040 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" Sep 14 11:29:06.435143 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.435059 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" event={"ID":"1feb9dd5-e808-4402-a88f-6c5726560ac4","Type":"ContainerDied","Data":"01ae59f1fc273123bd58f635aa3a915bb3ae66b653cb59cbea5127704b0e1958"} Sep 14 11:29:06.435143 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.435114 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw" event={"ID":"1feb9dd5-e808-4402-a88f-6c5726560ac4","Type":"ContainerDied","Data":"3c2175fc6ba50a0d491230aeaa829e9536d9c1c9bf70ae485d40c44da12053b5"} Sep 14 11:29:06.451279 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.451254 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw"] Sep 14 11:29:06.455325 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.455305 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/distinct-2-w0-6f3d0f-kserve-798f94fbd9-9jspw"] Sep 14 11:29:06.465729 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.465708 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2"] Sep 14 11:29:06.470185 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.470165 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/premium-w0-6f3d0f-kserve-684cd8899-ztgg2"] Sep 14 11:29:06.479223 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.479201 2556 scope.go:117] "RemoveContainer" containerID="6d17f337dc1c494799da4929a8bf6a7dc95e9cc7c17f9898864c64372ca383f5" Sep 14 11:29:06.479644 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:29:06.479624 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d17f337dc1c494799da4929a8bf6a7dc95e9cc7c17f9898864c64372ca383f5\": container with ID starting with 6d17f337dc1c494799da4929a8bf6a7dc95e9cc7c17f9898864c64372ca383f5 not found: ID does not exist" containerID="6d17f337dc1c494799da4929a8bf6a7dc95e9cc7c17f9898864c64372ca383f5" Sep 14 11:29:06.479720 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.479655 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d17f337dc1c494799da4929a8bf6a7dc95e9cc7c17f9898864c64372ca383f5"} err="failed to get container status \"6d17f337dc1c494799da4929a8bf6a7dc95e9cc7c17f9898864c64372ca383f5\": rpc error: code = NotFound desc = could not find container \"6d17f337dc1c494799da4929a8bf6a7dc95e9cc7c17f9898864c64372ca383f5\": container with ID starting with 6d17f337dc1c494799da4929a8bf6a7dc95e9cc7c17f9898864c64372ca383f5 not found: ID does not exist" Sep 14 11:29:06.479720 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.479679 2556 scope.go:117] "RemoveContainer" containerID="01ae59f1fc273123bd58f635aa3a915bb3ae66b653cb59cbea5127704b0e1958" Sep 14 11:29:06.488196 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.488178 2556 scope.go:117] "RemoveContainer" containerID="01ae59f1fc273123bd58f635aa3a915bb3ae66b653cb59cbea5127704b0e1958" Sep 14 11:29:06.488472 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:29:06.488455 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01ae59f1fc273123bd58f635aa3a915bb3ae66b653cb59cbea5127704b0e1958\": container with ID starting with 01ae59f1fc273123bd58f635aa3a915bb3ae66b653cb59cbea5127704b0e1958 not found: ID does not exist" containerID="01ae59f1fc273123bd58f635aa3a915bb3ae66b653cb59cbea5127704b0e1958" Sep 14 11:29:06.488532 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.488477 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01ae59f1fc273123bd58f635aa3a915bb3ae66b653cb59cbea5127704b0e1958"} err="failed to get container status \"01ae59f1fc273123bd58f635aa3a915bb3ae66b653cb59cbea5127704b0e1958\": rpc error: code = NotFound desc = could not find container \"01ae59f1fc273123bd58f635aa3a915bb3ae66b653cb59cbea5127704b0e1958\": container with ID starting with 01ae59f1fc273123bd58f635aa3a915bb3ae66b653cb59cbea5127704b0e1958 not found: ID does not exist" Sep 14 11:29:06.608558 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.608529 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:29:06.658205 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.658179 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mscg\" (UniqueName: \"kubernetes.io/projected/3a6f0275-dd16-4161-a0cf-d5ade98729d3-kube-api-access-6mscg\") pod \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\" (UID: \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\") " Sep 14 11:29:06.658329 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.658263 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a6f0275-dd16-4161-a0cf-d5ade98729d3-tmp-dir\") pod \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\" (UID: \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\") " Sep 14 11:29:06.658397 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.658353 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a6f0275-dd16-4161-a0cf-d5ade98729d3-tls-certs\") pod \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\" (UID: \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\") " Sep 14 11:29:06.658454 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.658398 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a6f0275-dd16-4161-a0cf-d5ade98729d3-dshm\") pod \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\" (UID: \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\") " Sep 14 11:29:06.658454 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.658433 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a6f0275-dd16-4161-a0cf-d5ade98729d3-home\") pod \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\" (UID: \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\") " Sep 14 11:29:06.658560 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.658500 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a6f0275-dd16-4161-a0cf-d5ade98729d3-model-cache\") pod \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\" (UID: \"3a6f0275-dd16-4161-a0cf-d5ade98729d3\") " Sep 14 11:29:06.658560 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.658539 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6f0275-dd16-4161-a0cf-d5ade98729d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3a6f0275-dd16-4161-a0cf-d5ade98729d3" (UID: "3a6f0275-dd16-4161-a0cf-d5ade98729d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:06.658948 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.658917 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a6f0275-dd16-4161-a0cf-d5ade98729d3-tmp-dir\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:06.659044 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.658957 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6f0275-dd16-4161-a0cf-d5ade98729d3-home" (OuterVolumeSpecName: "home") pod "3a6f0275-dd16-4161-a0cf-d5ade98729d3" (UID: "3a6f0275-dd16-4161-a0cf-d5ade98729d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:06.659132 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.659111 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6f0275-dd16-4161-a0cf-d5ade98729d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "3a6f0275-dd16-4161-a0cf-d5ade98729d3" (UID: "3a6f0275-dd16-4161-a0cf-d5ade98729d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:06.660826 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.660799 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a6f0275-dd16-4161-a0cf-d5ade98729d3-kube-api-access-6mscg" (OuterVolumeSpecName: "kube-api-access-6mscg") pod "3a6f0275-dd16-4161-a0cf-d5ade98729d3" (UID: "3a6f0275-dd16-4161-a0cf-d5ade98729d3"). InnerVolumeSpecName "kube-api-access-6mscg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:29:06.661330 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.660902 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a6f0275-dd16-4161-a0cf-d5ade98729d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a6f0275-dd16-4161-a0cf-d5ade98729d3" (UID: "3a6f0275-dd16-4161-a0cf-d5ade98729d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:29:06.661567 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.661539 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6f0275-dd16-4161-a0cf-d5ade98729d3-dshm" (OuterVolumeSpecName: "dshm") pod "3a6f0275-dd16-4161-a0cf-d5ade98729d3" (UID: "3a6f0275-dd16-4161-a0cf-d5ade98729d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:06.759879 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.759806 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a6f0275-dd16-4161-a0cf-d5ade98729d3-model-cache\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:06.759879 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.759831 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mscg\" (UniqueName: \"kubernetes.io/projected/3a6f0275-dd16-4161-a0cf-d5ade98729d3-kube-api-access-6mscg\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:06.759879 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.759842 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a6f0275-dd16-4161-a0cf-d5ade98729d3-tls-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:06.759879 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.759852 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a6f0275-dd16-4161-a0cf-d5ade98729d3-dshm\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:06.759879 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:06.759860 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a6f0275-dd16-4161-a0cf-d5ade98729d3-home\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:07.443620 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:07.443588 2556 generic.go:356] "Generic (PLEG): container finished" podID="3a6f0275-dd16-4161-a0cf-d5ade98729d3" containerID="6a9deab21e246b249ca5fad39d03e00c55ad92177368be1978068a80593eabb4" exitCode=0 Sep 14 11:29:07.443787 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:07.443626 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" Sep 14 11:29:07.443787 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:07.443679 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" event={"ID":"3a6f0275-dd16-4161-a0cf-d5ade98729d3","Type":"ContainerDied","Data":"6a9deab21e246b249ca5fad39d03e00c55ad92177368be1978068a80593eabb4"} Sep 14 11:29:07.443787 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:07.443724 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng" event={"ID":"3a6f0275-dd16-4161-a0cf-d5ade98729d3","Type":"ContainerDied","Data":"9cae18dd628d5f28fdc933fe917d3a74c78188a76895226a9dc7c0abdecef4f7"} Sep 14 11:29:07.443787 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:07.443745 2556 scope.go:117] "RemoveContainer" containerID="6a9deab21e246b249ca5fad39d03e00c55ad92177368be1978068a80593eabb4" Sep 14 11:29:07.452866 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:07.452849 2556 scope.go:117] "RemoveContainer" containerID="6a9deab21e246b249ca5fad39d03e00c55ad92177368be1978068a80593eabb4" Sep 14 11:29:07.453136 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:29:07.453117 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a9deab21e246b249ca5fad39d03e00c55ad92177368be1978068a80593eabb4\": container with ID starting with 6a9deab21e246b249ca5fad39d03e00c55ad92177368be1978068a80593eabb4 not found: ID does not exist" containerID="6a9deab21e246b249ca5fad39d03e00c55ad92177368be1978068a80593eabb4" Sep 14 11:29:07.453197 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:07.453145 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a9deab21e246b249ca5fad39d03e00c55ad92177368be1978068a80593eabb4"} err="failed to get container status \"6a9deab21e246b249ca5fad39d03e00c55ad92177368be1978068a80593eabb4\": rpc error: code = NotFound desc = could not find container \"6a9deab21e246b249ca5fad39d03e00c55ad92177368be1978068a80593eabb4\": container with ID starting with 6a9deab21e246b249ca5fad39d03e00c55ad92177368be1978068a80593eabb4 not found: ID does not exist" Sep 14 11:29:07.465423 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:07.465403 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng"] Sep 14 11:29:07.473717 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:07.473696 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/embedding-w0-6f3d0f-kserve-7bb6d67647-4wpng"] Sep 14 11:29:08.382670 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:08.382641 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1feb9dd5-e808-4402-a88f-6c5726560ac4" path="/var/lib/kubelet/pods/1feb9dd5-e808-4402-a88f-6c5726560ac4/volumes" Sep 14 11:29:08.383057 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:08.383017 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a6f0275-dd16-4161-a0cf-d5ade98729d3" path="/var/lib/kubelet/pods/3a6f0275-dd16-4161-a0cf-d5ade98729d3/volumes" Sep 14 11:29:08.383393 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:08.383380 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4138f69-d4ca-4f25-ae4f-4e889bf8a972" path="/var/lib/kubelet/pods/f4138f69-d4ca-4f25-ae4f-4e889bf8a972/volumes" Sep 14 11:29:11.408529 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:11.408496 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65" Sep 14 11:29:12.661446 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:12.661396 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl"] Sep 14 11:29:12.661894 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:12.661606 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl" podUID="ef1013ed-ced5-49c7-b743-8f4c587222a6" containerName="payload-pre-processing" containerID="cri-o://8a5d11fa1a47ae1ab6371fd824bc6bea88e6586d5c724f15bec75e8f089c26c2" gracePeriod=30 Sep 14 11:29:12.859571 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:12.859511 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4fc87-jz46g"] Sep 14 11:29:12.861827 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:12.861801 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4fc87-jz46g"] Sep 14 11:29:12.913788 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:12.913728 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl" Sep 14 11:29:13.020709 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:13.020681 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1013ed-ced5-49c7-b743-8f4c587222a6-plugins-config-volume\") pod \"ef1013ed-ced5-49c7-b743-8f4c587222a6\" (UID: \"ef1013ed-ced5-49c7-b743-8f4c587222a6\") " Sep 14 11:29:13.020888 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:13.020870 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwjbw\" (UniqueName: \"kubernetes.io/projected/ef1013ed-ced5-49c7-b743-8f4c587222a6-kube-api-access-lwjbw\") pod \"ef1013ed-ced5-49c7-b743-8f4c587222a6\" (UID: \"ef1013ed-ced5-49c7-b743-8f4c587222a6\") " Sep 14 11:29:13.021119 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:13.021081 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef1013ed-ced5-49c7-b743-8f4c587222a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ef1013ed-ced5-49c7-b743-8f4c587222a6" (UID: "ef1013ed-ced5-49c7-b743-8f4c587222a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:29:13.021262 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:13.021247 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1013ed-ced5-49c7-b743-8f4c587222a6-plugins-config-volume\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:13.022831 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:13.022813 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1013ed-ced5-49c7-b743-8f4c587222a6-kube-api-access-lwjbw" (OuterVolumeSpecName: "kube-api-access-lwjbw") pod "ef1013ed-ced5-49c7-b743-8f4c587222a6" (UID: "ef1013ed-ced5-49c7-b743-8f4c587222a6"). InnerVolumeSpecName "kube-api-access-lwjbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:29:13.122534 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:13.122504 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwjbw\" (UniqueName: \"kubernetes.io/projected/ef1013ed-ced5-49c7-b743-8f4c587222a6-kube-api-access-lwjbw\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:13.306039 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:13.305963 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g"] Sep 14 11:29:13.306210 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:13.306184 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" podUID="0b95e55c-aadc-440b-af1e-ce128f283433" containerName="istio-proxy" containerID="cri-o://2e9e8bb671637d334bae5b40eaad04fc53c5978a587459be11a37287d680d82c" gracePeriod=30 Sep 14 11:29:13.478430 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:13.478393 2556 generic.go:356] "Generic (PLEG): container finished" podID="ef1013ed-ced5-49c7-b743-8f4c587222a6" containerID="8a5d11fa1a47ae1ab6371fd824bc6bea88e6586d5c724f15bec75e8f089c26c2" exitCode=0 Sep 14 11:29:13.478606 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:13.478446 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl" Sep 14 11:29:13.478606 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:13.478481 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl" event={"ID":"ef1013ed-ced5-49c7-b743-8f4c587222a6","Type":"ContainerDied","Data":"8a5d11fa1a47ae1ab6371fd824bc6bea88e6586d5c724f15bec75e8f089c26c2"} Sep 14 11:29:13.478606 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:13.478544 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl" event={"ID":"ef1013ed-ced5-49c7-b743-8f4c587222a6","Type":"ContainerDied","Data":"486f15518f2328a62befee7112244221d590d2b90ab128d142b3077ac54d13ad"} Sep 14 11:29:13.478606 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:13.478576 2556 scope.go:117] "RemoveContainer" containerID="8a5d11fa1a47ae1ab6371fd824bc6bea88e6586d5c724f15bec75e8f089c26c2" Sep 14 11:29:13.489259 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:13.489155 2556 scope.go:117] "RemoveContainer" containerID="8a5d11fa1a47ae1ab6371fd824bc6bea88e6586d5c724f15bec75e8f089c26c2" Sep 14 11:29:13.489521 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:29:13.489493 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a5d11fa1a47ae1ab6371fd824bc6bea88e6586d5c724f15bec75e8f089c26c2\": container with ID starting with 8a5d11fa1a47ae1ab6371fd824bc6bea88e6586d5c724f15bec75e8f089c26c2 not found: ID does not exist" containerID="8a5d11fa1a47ae1ab6371fd824bc6bea88e6586d5c724f15bec75e8f089c26c2" Sep 14 11:29:13.489687 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:13.489658 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a5d11fa1a47ae1ab6371fd824bc6bea88e6586d5c724f15bec75e8f089c26c2"} err="failed to get container status \"8a5d11fa1a47ae1ab6371fd824bc6bea88e6586d5c724f15bec75e8f089c26c2\": rpc error: code = NotFound desc = could not find container \"8a5d11fa1a47ae1ab6371fd824bc6bea88e6586d5c724f15bec75e8f089c26c2\": container with ID starting with 8a5d11fa1a47ae1ab6371fd824bc6bea88e6586d5c724f15bec75e8f089c26c2 not found: ID does not exist" Sep 14 11:29:13.511640 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:13.511610 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl"] Sep 14 11:29:13.515322 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:13.515298 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e4fc87-84645566fd-kg7rl"] Sep 14 11:29:14.382452 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:14.382408 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa44a509-1e67-4811-b87f-5107e7ca41be" path="/var/lib/kubelet/pods/aa44a509-1e67-4811-b87f-5107e7ca41be/volumes" Sep 14 11:29:14.382975 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:14.382954 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef1013ed-ced5-49c7-b743-8f4c587222a6" path="/var/lib/kubelet/pods/ef1013ed-ced5-49c7-b743-8f4c587222a6/volumes" Sep 14 11:29:18.509134 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.509106 2556 generic.go:356] "Generic (PLEG): container finished" podID="0b95e55c-aadc-440b-af1e-ce128f283433" containerID="2e9e8bb671637d334bae5b40eaad04fc53c5978a587459be11a37287d680d82c" exitCode=0 Sep 14 11:29:18.509134 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.509118 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" event={"ID":"0b95e55c-aadc-440b-af1e-ce128f283433","Type":"ContainerDied","Data":"2e9e8bb671637d334bae5b40eaad04fc53c5978a587459be11a37287d680d82c"} Sep 14 11:29:18.509558 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.509180 2556 scope.go:117] "RemoveContainer" containerID="b2cc998f408459b66ac6db3633cffe9f711be7c6e06c20fbf8468065c734d9b3" Sep 14 11:29:18.550840 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.550822 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:29:18.675819 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.675785 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-istio-envoy\") pod \"0b95e55c-aadc-440b-af1e-ce128f283433\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " Sep 14 11:29:18.675952 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.675828 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-credential-socket\") pod \"0b95e55c-aadc-440b-af1e-ce128f283433\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " Sep 14 11:29:18.675952 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.675858 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b95e55c-aadc-440b-af1e-ce128f283433-istiod-ca-cert\") pod \"0b95e55c-aadc-440b-af1e-ce128f283433\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " Sep 14 11:29:18.675952 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.675911 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b95e55c-aadc-440b-af1e-ce128f283433-istio-token\") pod \"0b95e55c-aadc-440b-af1e-ce128f283433\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " Sep 14 11:29:18.675952 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.675939 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlqf4\" (UniqueName: \"kubernetes.io/projected/0b95e55c-aadc-440b-af1e-ce128f283433-kube-api-access-tlqf4\") pod \"0b95e55c-aadc-440b-af1e-ce128f283433\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " Sep 14 11:29:18.676201 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.676123 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-istio-data\") pod \"0b95e55c-aadc-440b-af1e-ce128f283433\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " Sep 14 11:29:18.676259 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.676206 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0b95e55c-aadc-440b-af1e-ce128f283433" (UID: "0b95e55c-aadc-440b-af1e-ce128f283433"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:18.676259 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.676228 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-workload-socket\") pod \"0b95e55c-aadc-440b-af1e-ce128f283433\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " Sep 14 11:29:18.676365 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.676259 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b95e55c-aadc-440b-af1e-ce128f283433-istio-podinfo\") pod \"0b95e55c-aadc-440b-af1e-ce128f283433\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " Sep 14 11:29:18.676365 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.676285 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b95e55c-aadc-440b-af1e-ce128f283433-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0b95e55c-aadc-440b-af1e-ce128f283433" (UID: "0b95e55c-aadc-440b-af1e-ce128f283433"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:29:18.676463 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.676334 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-istio-data" (OuterVolumeSpecName: "istio-data") pod "0b95e55c-aadc-440b-af1e-ce128f283433" (UID: "0b95e55c-aadc-440b-af1e-ce128f283433"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:18.676463 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.676294 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-workload-certs\") pod \"0b95e55c-aadc-440b-af1e-ce128f283433\" (UID: \"0b95e55c-aadc-440b-af1e-ce128f283433\") " Sep 14 11:29:18.676563 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.676489 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0b95e55c-aadc-440b-af1e-ce128f283433" (UID: "0b95e55c-aadc-440b-af1e-ce128f283433"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:18.676563 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.676536 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0b95e55c-aadc-440b-af1e-ce128f283433" (UID: "0b95e55c-aadc-440b-af1e-ce128f283433"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:18.676773 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.676751 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:18.676839 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.676779 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:18.676839 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.676794 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:18.676839 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.676808 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:18.676839 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.676820 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b95e55c-aadc-440b-af1e-ce128f283433-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:18.678073 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.678052 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0b95e55c-aadc-440b-af1e-ce128f283433" (UID: "0b95e55c-aadc-440b-af1e-ce128f283433"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:18.678223 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.678207 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b95e55c-aadc-440b-af1e-ce128f283433-istio-token" (OuterVolumeSpecName: "istio-token") pod "0b95e55c-aadc-440b-af1e-ce128f283433" (UID: "0b95e55c-aadc-440b-af1e-ce128f283433"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:29:18.678292 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.678271 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b95e55c-aadc-440b-af1e-ce128f283433-kube-api-access-tlqf4" (OuterVolumeSpecName: "kube-api-access-tlqf4") pod "0b95e55c-aadc-440b-af1e-ce128f283433" (UID: "0b95e55c-aadc-440b-af1e-ce128f283433"). InnerVolumeSpecName "kube-api-access-tlqf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:29:18.678518 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.678500 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b95e55c-aadc-440b-af1e-ce128f283433-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0b95e55c-aadc-440b-af1e-ce128f283433" (UID: "0b95e55c-aadc-440b-af1e-ce128f283433"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 11:29:18.777263 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.777224 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlqf4\" (UniqueName: \"kubernetes.io/projected/0b95e55c-aadc-440b-af1e-ce128f283433-kube-api-access-tlqf4\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:18.777263 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.777260 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b95e55c-aadc-440b-af1e-ce128f283433-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:18.777396 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.777274 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b95e55c-aadc-440b-af1e-ce128f283433-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:18.777396 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:18.777285 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b95e55c-aadc-440b-af1e-ce128f283433-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:19.522368 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:19.522334 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" event={"ID":"0b95e55c-aadc-440b-af1e-ce128f283433","Type":"ContainerDied","Data":"dcae9f88a136896dab8d90ae42051ceebd634de957520fbf24edbf3213adf959"} Sep 14 11:29:19.522798 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:19.522374 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g" Sep 14 11:29:19.522798 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:19.522429 2556 scope.go:117] "RemoveContainer" containerID="2e9e8bb671637d334bae5b40eaad04fc53c5978a587459be11a37287d680d82c" Sep 14 11:29:19.545024 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:19.545000 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g"] Sep 14 11:29:19.548476 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:19.548455 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6f3d0f-openshift-default-6bff784866-fjn9g"] Sep 14 11:29:19.824055 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:19.823979 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq"] Sep 14 11:29:19.824265 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:19.824239 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" podUID="4359bf25-5e49-4a8d-92b5-5fd66271b51f" containerName="istio-proxy" containerID="cri-o://27d67f26e3976c40a771838b2ce00b6f4a162cdba9c5fea1093324f89a9c057e" gracePeriod=30 Sep 14 11:29:20.115922 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.115884 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-420d64-2m4j9"] Sep 14 11:29:20.116332 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.116313 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b95e55c-aadc-440b-af1e-ce128f283433" containerName="istio-proxy" Sep 14 11:29:20.116424 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.116334 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b95e55c-aadc-440b-af1e-ce128f283433" containerName="istio-proxy" Sep 14 11:29:20.116424 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.116346 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa44a509-1e67-4811-b87f-5107e7ca41be" containerName="revoke-keys" Sep 14 11:29:20.116424 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.116353 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa44a509-1e67-4811-b87f-5107e7ca41be" containerName="revoke-keys" Sep 14 11:29:20.116424 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.116372 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef1013ed-ced5-49c7-b743-8f4c587222a6" containerName="payload-pre-processing" Sep 14 11:29:20.116424 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.116380 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef1013ed-ced5-49c7-b743-8f4c587222a6" containerName="payload-pre-processing" Sep 14 11:29:20.116424 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.116394 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4138f69-d4ca-4f25-ae4f-4e889bf8a972" containerName="main" Sep 14 11:29:20.116424 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.116402 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4138f69-d4ca-4f25-ae4f-4e889bf8a972" containerName="main" Sep 14 11:29:20.116424 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.116427 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1feb9dd5-e808-4402-a88f-6c5726560ac4" containerName="main" Sep 14 11:29:20.116857 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.116435 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1feb9dd5-e808-4402-a88f-6c5726560ac4" containerName="main" Sep 14 11:29:20.116857 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.116449 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a6f0275-dd16-4161-a0cf-d5ade98729d3" containerName="main" Sep 14 11:29:20.116857 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.116458 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a6f0275-dd16-4161-a0cf-d5ade98729d3" containerName="main" Sep 14 11:29:20.116857 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.116549 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a6f0275-dd16-4161-a0cf-d5ade98729d3" containerName="main" Sep 14 11:29:20.116857 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.116560 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4138f69-d4ca-4f25-ae4f-4e889bf8a972" containerName="main" Sep 14 11:29:20.116857 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.116571 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1feb9dd5-e808-4402-a88f-6c5726560ac4" containerName="main" Sep 14 11:29:20.116857 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.116581 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b95e55c-aadc-440b-af1e-ce128f283433" containerName="istio-proxy" Sep 14 11:29:20.116857 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.116591 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa44a509-1e67-4811-b87f-5107e7ca41be" containerName="revoke-keys" Sep 14 11:29:20.116857 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.116603 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef1013ed-ced5-49c7-b743-8f4c587222a6" containerName="payload-pre-processing" Sep 14 11:29:20.116857 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.116612 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b95e55c-aadc-440b-af1e-ce128f283433" containerName="istio-proxy" Sep 14 11:29:20.120984 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.120964 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-420d64-2m4j9" Sep 14 11:29:20.123412 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.123389 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f98vt\"" Sep 14 11:29:20.123412 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.123401 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 11:29:20.128325 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.128034 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-420d64-2m4j9"] Sep 14 11:29:20.193033 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.192997 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52b7c187-9461-45a7-a43b-e6f801116de7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-420d64-2m4j9\" (UID: \"52b7c187-9461-45a7-a43b-e6f801116de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-420d64-2m4j9" Sep 14 11:29:20.293651 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.293619 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52b7c187-9461-45a7-a43b-e6f801116de7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-420d64-2m4j9\" (UID: \"52b7c187-9461-45a7-a43b-e6f801116de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-420d64-2m4j9" Sep 14 11:29:20.294239 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.294220 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52b7c187-9461-45a7-a43b-e6f801116de7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-420d64-2m4j9\" (UID: \"52b7c187-9461-45a7-a43b-e6f801116de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-420d64-2m4j9" Sep 14 11:29:20.383047 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.382971 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b95e55c-aadc-440b-af1e-ce128f283433" path="/var/lib/kubelet/pods/0b95e55c-aadc-440b-af1e-ce128f283433/volumes" Sep 14 11:29:20.433581 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.433557 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-420d64-2m4j9" Sep 14 11:29:20.709561 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:20.709541 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-420d64-2m4j9"] Sep 14 11:29:20.711262 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:29:20.711237 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b7c187_9461_45a7_a43b_e6f801116de7.slice/crio-ab66491c77b96775c7b7c172d4cd89ce71df19fe55f75d1f97189ae9c4192ce6 WatchSource:0}: Error finding container ab66491c77b96775c7b7c172d4cd89ce71df19fe55f75d1f97189ae9c4192ce6: Status 404 returned error can't find the container with id ab66491c77b96775c7b7c172d4cd89ce71df19fe55f75d1f97189ae9c4192ce6 Sep 14 11:29:21.535734 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:21.535702 2556 generic.go:356] "Generic (PLEG): container finished" podID="52b7c187-9461-45a7-a43b-e6f801116de7" containerID="085f46519a8179cdeca475cbaaa328ccd56b4d5686a9048408e448d3db58beef" exitCode=0 Sep 14 11:29:21.535876 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:21.535752 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-420d64-2m4j9" event={"ID":"52b7c187-9461-45a7-a43b-e6f801116de7","Type":"ContainerDied","Data":"085f46519a8179cdeca475cbaaa328ccd56b4d5686a9048408e448d3db58beef"} Sep 14 11:29:21.535876 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:21.535791 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-420d64-2m4j9" event={"ID":"52b7c187-9461-45a7-a43b-e6f801116de7","Type":"ContainerStarted","Data":"ab66491c77b96775c7b7c172d4cd89ce71df19fe55f75d1f97189ae9c4192ce6"} Sep 14 11:29:22.678416 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:22.678391 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-420d64-2m4j9" Sep 14 11:29:22.820298 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:22.820235 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52b7c187-9461-45a7-a43b-e6f801116de7-maas-api-service-ca\") pod \"52b7c187-9461-45a7-a43b-e6f801116de7\" (UID: \"52b7c187-9461-45a7-a43b-e6f801116de7\") " Sep 14 11:29:22.820560 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:22.820540 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b7c187-9461-45a7-a43b-e6f801116de7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "52b7c187-9461-45a7-a43b-e6f801116de7" (UID: "52b7c187-9461-45a7-a43b-e6f801116de7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:29:22.920918 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:22.920887 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52b7c187-9461-45a7-a43b-e6f801116de7-maas-api-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:23.547334 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:23.547305 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-420d64-2m4j9" Sep 14 11:29:23.547494 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:23.547347 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-420d64-2m4j9" event={"ID":"52b7c187-9461-45a7-a43b-e6f801116de7","Type":"ContainerDied","Data":"ab66491c77b96775c7b7c172d4cd89ce71df19fe55f75d1f97189ae9c4192ce6"} Sep 14 11:29:23.547494 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:23.547394 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab66491c77b96775c7b7c172d4cd89ce71df19fe55f75d1f97189ae9c4192ce6" Sep 14 11:29:25.075325 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.075300 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:29:25.249663 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.249634 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ktjt\" (UniqueName: \"kubernetes.io/projected/4359bf25-5e49-4a8d-92b5-5fd66271b51f-kube-api-access-7ktjt\") pod \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " Sep 14 11:29:25.249663 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.249666 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-credential-socket\") pod \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " Sep 14 11:29:25.249867 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.249697 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-workload-socket\") pod \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " Sep 14 11:29:25.249867 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.249724 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istio-data\") pod \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " Sep 14 11:29:25.249867 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.249749 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-workload-certs\") pod \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " Sep 14 11:29:25.249867 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.249785 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istio-envoy\") pod \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " Sep 14 11:29:25.249867 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.249857 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istio-podinfo\") pod \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " Sep 14 11:29:25.250137 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.249894 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istio-token\") pod \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " Sep 14 11:29:25.250137 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.249936 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4359bf25-5e49-4a8d-92b5-5fd66271b51f" (UID: "4359bf25-5e49-4a8d-92b5-5fd66271b51f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:25.250137 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.249956 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istiod-ca-cert\") pod \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\" (UID: \"4359bf25-5e49-4a8d-92b5-5fd66271b51f\") " Sep 14 11:29:25.250137 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.250022 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istio-data" (OuterVolumeSpecName: "istio-data") pod "4359bf25-5e49-4a8d-92b5-5fd66271b51f" (UID: "4359bf25-5e49-4a8d-92b5-5fd66271b51f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:25.250137 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.250041 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4359bf25-5e49-4a8d-92b5-5fd66271b51f" (UID: "4359bf25-5e49-4a8d-92b5-5fd66271b51f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:25.250388 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.250354 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:25.250388 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.250361 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4359bf25-5e49-4a8d-92b5-5fd66271b51f" (UID: "4359bf25-5e49-4a8d-92b5-5fd66271b51f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:29:25.250388 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.250374 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:25.250388 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.250388 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:25.250586 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.250422 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4359bf25-5e49-4a8d-92b5-5fd66271b51f" (UID: "4359bf25-5e49-4a8d-92b5-5fd66271b51f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:25.252079 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.252054 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4359bf25-5e49-4a8d-92b5-5fd66271b51f" (UID: "4359bf25-5e49-4a8d-92b5-5fd66271b51f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 11:29:25.252215 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.252053 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4359bf25-5e49-4a8d-92b5-5fd66271b51f" (UID: "4359bf25-5e49-4a8d-92b5-5fd66271b51f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:25.252294 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.252258 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4359bf25-5e49-4a8d-92b5-5fd66271b51f-kube-api-access-7ktjt" (OuterVolumeSpecName: "kube-api-access-7ktjt") pod "4359bf25-5e49-4a8d-92b5-5fd66271b51f" (UID: "4359bf25-5e49-4a8d-92b5-5fd66271b51f"). InnerVolumeSpecName "kube-api-access-7ktjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:29:25.252294 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.252265 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istio-token" (OuterVolumeSpecName: "istio-token") pod "4359bf25-5e49-4a8d-92b5-5fd66271b51f" (UID: "4359bf25-5e49-4a8d-92b5-5fd66271b51f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:29:25.351528 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.351492 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:25.351528 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.351522 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ktjt\" (UniqueName: \"kubernetes.io/projected/4359bf25-5e49-4a8d-92b5-5fd66271b51f-kube-api-access-7ktjt\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:25.351528 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.351533 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:25.351756 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.351541 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:25.351756 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.351549 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:25.351756 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.351557 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4359bf25-5e49-4a8d-92b5-5fd66271b51f-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:25.560360 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.560284 2556 generic.go:356] "Generic (PLEG): container finished" podID="4359bf25-5e49-4a8d-92b5-5fd66271b51f" containerID="27d67f26e3976c40a771838b2ce00b6f4a162cdba9c5fea1093324f89a9c057e" exitCode=0 Sep 14 11:29:25.560360 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.560329 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" Sep 14 11:29:25.560525 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.560367 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" event={"ID":"4359bf25-5e49-4a8d-92b5-5fd66271b51f","Type":"ContainerDied","Data":"27d67f26e3976c40a771838b2ce00b6f4a162cdba9c5fea1093324f89a9c057e"} Sep 14 11:29:25.560525 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.560422 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq" event={"ID":"4359bf25-5e49-4a8d-92b5-5fd66271b51f","Type":"ContainerDied","Data":"260c19e6b93c440ce38d004d108d5dabb694aeef25cd77068318c06953bd4a4f"} Sep 14 11:29:25.560525 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.560444 2556 scope.go:117] "RemoveContainer" containerID="27d67f26e3976c40a771838b2ce00b6f4a162cdba9c5fea1093324f89a9c057e" Sep 14 11:29:25.569641 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.569626 2556 scope.go:117] "RemoveContainer" containerID="fe1ceec7f0acf65795476ac998cda9e1aeb32980cc1104de26532cf625e6d6ad" Sep 14 11:29:25.577153 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.577135 2556 scope.go:117] "RemoveContainer" containerID="27d67f26e3976c40a771838b2ce00b6f4a162cdba9c5fea1093324f89a9c057e" Sep 14 11:29:25.577406 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:29:25.577386 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27d67f26e3976c40a771838b2ce00b6f4a162cdba9c5fea1093324f89a9c057e\": container with ID starting with 27d67f26e3976c40a771838b2ce00b6f4a162cdba9c5fea1093324f89a9c057e not found: ID does not exist" containerID="27d67f26e3976c40a771838b2ce00b6f4a162cdba9c5fea1093324f89a9c057e" Sep 14 11:29:25.577447 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.577415 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d67f26e3976c40a771838b2ce00b6f4a162cdba9c5fea1093324f89a9c057e"} err="failed to get container status \"27d67f26e3976c40a771838b2ce00b6f4a162cdba9c5fea1093324f89a9c057e\": rpc error: code = NotFound desc = could not find container \"27d67f26e3976c40a771838b2ce00b6f4a162cdba9c5fea1093324f89a9c057e\": container with ID starting with 27d67f26e3976c40a771838b2ce00b6f4a162cdba9c5fea1093324f89a9c057e not found: ID does not exist" Sep 14 11:29:25.577447 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.577430 2556 scope.go:117] "RemoveContainer" containerID="fe1ceec7f0acf65795476ac998cda9e1aeb32980cc1104de26532cf625e6d6ad" Sep 14 11:29:25.577669 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:29:25.577651 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe1ceec7f0acf65795476ac998cda9e1aeb32980cc1104de26532cf625e6d6ad\": container with ID starting with fe1ceec7f0acf65795476ac998cda9e1aeb32980cc1104de26532cf625e6d6ad not found: ID does not exist" containerID="fe1ceec7f0acf65795476ac998cda9e1aeb32980cc1104de26532cf625e6d6ad" Sep 14 11:29:25.577720 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.577675 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe1ceec7f0acf65795476ac998cda9e1aeb32980cc1104de26532cf625e6d6ad"} err="failed to get container status \"fe1ceec7f0acf65795476ac998cda9e1aeb32980cc1104de26532cf625e6d6ad\": rpc error: code = NotFound desc = could not find container \"fe1ceec7f0acf65795476ac998cda9e1aeb32980cc1104de26532cf625e6d6ad\": container with ID starting with fe1ceec7f0acf65795476ac998cda9e1aeb32980cc1104de26532cf625e6d6ad not found: ID does not exist" Sep 14 11:29:25.587141 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.587113 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq"] Sep 14 11:29:25.590077 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:25.590054 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e4fc87-openshift-default-6d76867d-82lwq"] Sep 14 11:29:26.383920 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:26.383880 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4359bf25-5e49-4a8d-92b5-5fd66271b51f" path="/var/lib/kubelet/pods/4359bf25-5e49-4a8d-92b5-5fd66271b51f/volumes" Sep 14 11:29:36.052754 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:36.052718 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg"] Sep 14 11:29:36.053261 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:36.053002 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg" podUID="e0e612e7-2a2b-4d68-9e0c-d9b0151e4984" containerName="payload-processing" containerID="cri-o://4e24e24c710c287cbeeb6fde8fadd123d8e0204ad884eafca8fc103b8547a290" gracePeriod=30 Sep 14 11:29:36.230238 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:36.230186 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-420d64-2m4j9"] Sep 14 11:29:36.234030 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:36.233966 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-420d64-2m4j9"] Sep 14 11:29:36.321786 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:36.321764 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg" Sep 14 11:29:36.363782 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:36.363752 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mllj\" (UniqueName: \"kubernetes.io/projected/e0e612e7-2a2b-4d68-9e0c-d9b0151e4984-kube-api-access-6mllj\") pod \"e0e612e7-2a2b-4d68-9e0c-d9b0151e4984\" (UID: \"e0e612e7-2a2b-4d68-9e0c-d9b0151e4984\") " Sep 14 11:29:36.363954 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:36.363806 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e612e7-2a2b-4d68-9e0c-d9b0151e4984-plugins-config-volume\") pod \"e0e612e7-2a2b-4d68-9e0c-d9b0151e4984\" (UID: \"e0e612e7-2a2b-4d68-9e0c-d9b0151e4984\") " Sep 14 11:29:36.364236 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:36.364211 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e612e7-2a2b-4d68-9e0c-d9b0151e4984-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e0e612e7-2a2b-4d68-9e0c-d9b0151e4984" (UID: "e0e612e7-2a2b-4d68-9e0c-d9b0151e4984"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:29:36.365948 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:36.365917 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e612e7-2a2b-4d68-9e0c-d9b0151e4984-kube-api-access-6mllj" (OuterVolumeSpecName: "kube-api-access-6mllj") pod "e0e612e7-2a2b-4d68-9e0c-d9b0151e4984" (UID: "e0e612e7-2a2b-4d68-9e0c-d9b0151e4984"). InnerVolumeSpecName "kube-api-access-6mllj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:29:36.405112 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:36.401083 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52b7c187-9461-45a7-a43b-e6f801116de7" path="/var/lib/kubelet/pods/52b7c187-9461-45a7-a43b-e6f801116de7/volumes" Sep 14 11:29:36.465184 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:36.465140 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mllj\" (UniqueName: \"kubernetes.io/projected/e0e612e7-2a2b-4d68-9e0c-d9b0151e4984-kube-api-access-6mllj\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:36.465184 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:36.465179 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e612e7-2a2b-4d68-9e0c-d9b0151e4984-plugins-config-volume\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:36.630060 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:36.630024 2556 generic.go:356] "Generic (PLEG): container finished" podID="e0e612e7-2a2b-4d68-9e0c-d9b0151e4984" containerID="4e24e24c710c287cbeeb6fde8fadd123d8e0204ad884eafca8fc103b8547a290" exitCode=0 Sep 14 11:29:36.630265 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:36.630066 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg" Sep 14 11:29:36.630265 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:36.630127 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg" event={"ID":"e0e612e7-2a2b-4d68-9e0c-d9b0151e4984","Type":"ContainerDied","Data":"4e24e24c710c287cbeeb6fde8fadd123d8e0204ad884eafca8fc103b8547a290"} Sep 14 11:29:36.630265 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:36.630186 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg" event={"ID":"e0e612e7-2a2b-4d68-9e0c-d9b0151e4984","Type":"ContainerDied","Data":"c661a272d8cc93bc6d154d5c225f15538b85baa0d8c5c6fc2ba7ff3014077487"} Sep 14 11:29:36.630265 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:36.630215 2556 scope.go:117] "RemoveContainer" containerID="4e24e24c710c287cbeeb6fde8fadd123d8e0204ad884eafca8fc103b8547a290" Sep 14 11:29:36.639440 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:36.639422 2556 scope.go:117] "RemoveContainer" containerID="4e24e24c710c287cbeeb6fde8fadd123d8e0204ad884eafca8fc103b8547a290" Sep 14 11:29:36.639725 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:29:36.639707 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e24e24c710c287cbeeb6fde8fadd123d8e0204ad884eafca8fc103b8547a290\": container with ID starting with 4e24e24c710c287cbeeb6fde8fadd123d8e0204ad884eafca8fc103b8547a290 not found: ID does not exist" containerID="4e24e24c710c287cbeeb6fde8fadd123d8e0204ad884eafca8fc103b8547a290" Sep 14 11:29:36.639779 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:36.639735 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e24e24c710c287cbeeb6fde8fadd123d8e0204ad884eafca8fc103b8547a290"} err="failed to get container status \"4e24e24c710c287cbeeb6fde8fadd123d8e0204ad884eafca8fc103b8547a290\": rpc error: code = NotFound desc = could not find container \"4e24e24c710c287cbeeb6fde8fadd123d8e0204ad884eafca8fc103b8547a290\": container with ID starting with 4e24e24c710c287cbeeb6fde8fadd123d8e0204ad884eafca8fc103b8547a290 not found: ID does not exist" Sep 14 11:29:36.649821 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:36.649763 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg"] Sep 14 11:29:36.657905 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:36.657885 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-420d64-5dddf457f7-c76kg"] Sep 14 11:29:38.385917 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:38.385879 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0e612e7-2a2b-4d68-9e0c-d9b0151e4984" path="/var/lib/kubelet/pods/e0e612e7-2a2b-4d68-9e0c-d9b0151e4984/volumes" Sep 14 11:29:38.645438 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:38.645383 2556 generic.go:356] "Generic (PLEG): container finished" podID="d7d65916-67c9-42b7-8b55-3311ed30ab2d" containerID="23abcf0b22754cac9012819f0f6a038e73333c6d383f3c8a841f3ca357ab4933" exitCode=0 Sep 14 11:29:38.645566 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:38.645459 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" event={"ID":"d7d65916-67c9-42b7-8b55-3311ed30ab2d","Type":"ContainerDied","Data":"23abcf0b22754cac9012819f0f6a038e73333c6d383f3c8a841f3ca357ab4933"} Sep 14 11:29:38.645935 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:38.645918 2556 scope.go:117] "RemoveContainer" containerID="23abcf0b22754cac9012819f0f6a038e73333c6d383f3c8a841f3ca357ab4933" Sep 14 11:29:38.647048 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:38.647021 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:29:38.647117 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:38.647106 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:29:38.647173 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:38.647141 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:29:39.653146 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:39.653109 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" event={"ID":"d7d65916-67c9-42b7-8b55-3311ed30ab2d","Type":"ContainerStarted","Data":"6f90fda14f5a6d3496cc77a3dbcb83e40aef661f11fc598056c8a5962451271f"} Sep 14 11:29:40.072583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:40.072508 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:29:40.073500 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:40.073470 2556 patch_prober.go:28] interesting pod/e2e-api-b-420d64-openshift-default-5654854486-b5lst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 14 11:29:40.073632 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:40.073521 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" podUID="d7d65916-67c9-42b7-8b55-3311ed30ab2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 14 11:29:41.073141 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:41.073083 2556 patch_prober.go:28] interesting pod/e2e-api-b-420d64-openshift-default-5654854486-b5lst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 14 11:29:41.073588 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:41.073167 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" podUID="d7d65916-67c9-42b7-8b55-3311ed30ab2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 14 11:29:42.072110 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:42.072059 2556 patch_prober.go:28] interesting pod/e2e-api-b-420d64-openshift-default-5654854486-b5lst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 14 11:29:42.072277 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:42.072137 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" podUID="d7d65916-67c9-42b7-8b55-3311ed30ab2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 14 11:29:42.643593 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:42.643561 2556 scope.go:117] "RemoveContainer" containerID="488cc8fe175187f61fdffaf9574f0645668b98cf6bca86fc7c9920db3742012f" Sep 14 11:29:42.651682 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:42.651661 2556 scope.go:117] "RemoveContainer" containerID="732e083df2d1cb08cef3349fad1739accf68bad8b48f60a26e5185aa725c8dd9" Sep 14 11:29:43.072374 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.072289 2556 patch_prober.go:28] interesting pod/e2e-api-b-420d64-openshift-default-5654854486-b5lst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 14 11:29:43.072374 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.072345 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" podUID="d7d65916-67c9-42b7-8b55-3311ed30ab2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 14 11:29:43.126933 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.126898 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst"] Sep 14 11:29:43.127216 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.127187 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" podUID="d7d65916-67c9-42b7-8b55-3311ed30ab2d" containerName="istio-proxy" containerID="cri-o://6f90fda14f5a6d3496cc77a3dbcb83e40aef661f11fc598056c8a5962451271f" gracePeriod=30 Sep 14 11:29:43.214285 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.214230 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:29:43.499704 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.499664 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e42fe6-jqpjw"] Sep 14 11:29:43.500256 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.500233 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b95e55c-aadc-440b-af1e-ce128f283433" containerName="istio-proxy" Sep 14 11:29:43.500256 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.500257 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b95e55c-aadc-440b-af1e-ce128f283433" containerName="istio-proxy" Sep 14 11:29:43.500434 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.500278 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52b7c187-9461-45a7-a43b-e6f801116de7" containerName="revoke-keys" Sep 14 11:29:43.500434 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.500287 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="52b7c187-9461-45a7-a43b-e6f801116de7" containerName="revoke-keys" Sep 14 11:29:43.500434 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.500305 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0e612e7-2a2b-4d68-9e0c-d9b0151e4984" containerName="payload-processing" Sep 14 11:29:43.500434 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.500316 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0e612e7-2a2b-4d68-9e0c-d9b0151e4984" containerName="payload-processing" Sep 14 11:29:43.500434 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.500332 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4359bf25-5e49-4a8d-92b5-5fd66271b51f" containerName="istio-proxy" Sep 14 11:29:43.500434 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.500340 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4359bf25-5e49-4a8d-92b5-5fd66271b51f" containerName="istio-proxy" Sep 14 11:29:43.500434 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.500350 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4359bf25-5e49-4a8d-92b5-5fd66271b51f" containerName="istio-proxy" Sep 14 11:29:43.500434 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.500358 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4359bf25-5e49-4a8d-92b5-5fd66271b51f" containerName="istio-proxy" Sep 14 11:29:43.500852 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.500459 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="52b7c187-9461-45a7-a43b-e6f801116de7" containerName="revoke-keys" Sep 14 11:29:43.500852 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.500470 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4359bf25-5e49-4a8d-92b5-5fd66271b51f" containerName="istio-proxy" Sep 14 11:29:43.500852 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.500482 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0e612e7-2a2b-4d68-9e0c-d9b0151e4984" containerName="payload-processing" Sep 14 11:29:43.500852 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.500493 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4359bf25-5e49-4a8d-92b5-5fd66271b51f" containerName="istio-proxy" Sep 14 11:29:43.504789 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.504764 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e42fe6-jqpjw" Sep 14 11:29:43.507454 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.507392 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 11:29:43.507555 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.507491 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f98vt\"" Sep 14 11:29:43.518827 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.518806 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e42fe6-jqpjw"] Sep 14 11:29:43.532540 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.532485 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1ecb57-4773-43f6-aecd-a35aa7ebac43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e42fe6-jqpjw\" (UID: \"9e1ecb57-4773-43f6-aecd-a35aa7ebac43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e42fe6-jqpjw" Sep 14 11:29:43.633631 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.633603 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1ecb57-4773-43f6-aecd-a35aa7ebac43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e42fe6-jqpjw\" (UID: \"9e1ecb57-4773-43f6-aecd-a35aa7ebac43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e42fe6-jqpjw" Sep 14 11:29:43.634231 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.634214 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1ecb57-4773-43f6-aecd-a35aa7ebac43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e42fe6-jqpjw\" (UID: \"9e1ecb57-4773-43f6-aecd-a35aa7ebac43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e42fe6-jqpjw" Sep 14 11:29:43.819568 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:43.819495 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e42fe6-jqpjw" Sep 14 11:29:44.087749 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:44.087661 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e42fe6-jqpjw"] Sep 14 11:29:44.092616 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:29:44.092581 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1ecb57_4773_43f6_aecd_a35aa7ebac43.slice/crio-3f9d194099ae561162b1f9c0442cd6cf7cdc3cf66a0f283451cfd65c1e44a65a WatchSource:0}: Error finding container 3f9d194099ae561162b1f9c0442cd6cf7cdc3cf66a0f283451cfd65c1e44a65a: Status 404 returned error can't find the container with id 3f9d194099ae561162b1f9c0442cd6cf7cdc3cf66a0f283451cfd65c1e44a65a Sep 14 11:29:44.688120 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:44.688066 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e42fe6-jqpjw" event={"ID":"9e1ecb57-4773-43f6-aecd-a35aa7ebac43","Type":"ContainerStarted","Data":"b97ff7fbcdaf06fc0ff502661b6098b3e00424b69cd5eb09eebb9c3f2f4d0ac5"} Sep 14 11:29:44.688251 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:44.688147 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e42fe6-jqpjw" event={"ID":"9e1ecb57-4773-43f6-aecd-a35aa7ebac43","Type":"ContainerStarted","Data":"3f9d194099ae561162b1f9c0442cd6cf7cdc3cf66a0f283451cfd65c1e44a65a"} Sep 14 11:29:44.707386 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:44.707343 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e42fe6-jqpjw" podStartSLOduration=1.7073261770000001 podStartE2EDuration="1.707326177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:29:44.705508963 +0000 UTC m=+1742.768911874" watchObservedRunningTime="2026-09-14 11:29:44.707326177 +0000 UTC m=+1742.770729085" Sep 14 11:29:45.696839 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:45.696801 2556 generic.go:356] "Generic (PLEG): container finished" podID="9e1ecb57-4773-43f6-aecd-a35aa7ebac43" containerID="b97ff7fbcdaf06fc0ff502661b6098b3e00424b69cd5eb09eebb9c3f2f4d0ac5" exitCode=0 Sep 14 11:29:45.697309 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:45.696886 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e42fe6-jqpjw" event={"ID":"9e1ecb57-4773-43f6-aecd-a35aa7ebac43","Type":"ContainerDied","Data":"b97ff7fbcdaf06fc0ff502661b6098b3e00424b69cd5eb09eebb9c3f2f4d0ac5"} Sep 14 11:29:46.830372 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:46.830350 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e42fe6-jqpjw" Sep 14 11:29:46.867731 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:46.867696 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1ecb57-4773-43f6-aecd-a35aa7ebac43-maas-api-service-ca\") pod \"9e1ecb57-4773-43f6-aecd-a35aa7ebac43\" (UID: \"9e1ecb57-4773-43f6-aecd-a35aa7ebac43\") " Sep 14 11:29:46.868133 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:46.868077 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e1ecb57-4773-43f6-aecd-a35aa7ebac43-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9e1ecb57-4773-43f6-aecd-a35aa7ebac43" (UID: "9e1ecb57-4773-43f6-aecd-a35aa7ebac43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:29:46.968792 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:46.968713 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1ecb57-4773-43f6-aecd-a35aa7ebac43-maas-api-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:47.707764 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:47.707734 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e42fe6-jqpjw" event={"ID":"9e1ecb57-4773-43f6-aecd-a35aa7ebac43","Type":"ContainerDied","Data":"3f9d194099ae561162b1f9c0442cd6cf7cdc3cf66a0f283451cfd65c1e44a65a"} Sep 14 11:29:47.707764 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:47.707750 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e42fe6-jqpjw" Sep 14 11:29:47.707958 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:47.707777 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f9d194099ae561162b1f9c0442cd6cf7cdc3cf66a0f283451cfd65c1e44a65a" Sep 14 11:29:49.668810 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.668787 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:29:49.719080 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.719006 2556 generic.go:356] "Generic (PLEG): container finished" podID="d7d65916-67c9-42b7-8b55-3311ed30ab2d" containerID="6f90fda14f5a6d3496cc77a3dbcb83e40aef661f11fc598056c8a5962451271f" exitCode=0 Sep 14 11:29:49.719218 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.719076 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" Sep 14 11:29:49.719218 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.719111 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" event={"ID":"d7d65916-67c9-42b7-8b55-3311ed30ab2d","Type":"ContainerDied","Data":"6f90fda14f5a6d3496cc77a3dbcb83e40aef661f11fc598056c8a5962451271f"} Sep 14 11:29:49.719218 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.719177 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst" event={"ID":"d7d65916-67c9-42b7-8b55-3311ed30ab2d","Type":"ContainerDied","Data":"c66b77fadd7e31b720ab0c2bad5414f3cad13da5119a8c9f3de42e24911d5b88"} Sep 14 11:29:49.719218 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.719208 2556 scope.go:117] "RemoveContainer" containerID="6f90fda14f5a6d3496cc77a3dbcb83e40aef661f11fc598056c8a5962451271f" Sep 14 11:29:49.727487 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.727470 2556 scope.go:117] "RemoveContainer" containerID="23abcf0b22754cac9012819f0f6a038e73333c6d383f3c8a841f3ca357ab4933" Sep 14 11:29:49.734996 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.734981 2556 scope.go:117] "RemoveContainer" containerID="6f90fda14f5a6d3496cc77a3dbcb83e40aef661f11fc598056c8a5962451271f" Sep 14 11:29:49.735285 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:29:49.735267 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f90fda14f5a6d3496cc77a3dbcb83e40aef661f11fc598056c8a5962451271f\": container with ID starting with 6f90fda14f5a6d3496cc77a3dbcb83e40aef661f11fc598056c8a5962451271f not found: ID does not exist" containerID="6f90fda14f5a6d3496cc77a3dbcb83e40aef661f11fc598056c8a5962451271f" Sep 14 11:29:49.735337 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.735295 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f90fda14f5a6d3496cc77a3dbcb83e40aef661f11fc598056c8a5962451271f"} err="failed to get container status \"6f90fda14f5a6d3496cc77a3dbcb83e40aef661f11fc598056c8a5962451271f\": rpc error: code = NotFound desc = could not find container \"6f90fda14f5a6d3496cc77a3dbcb83e40aef661f11fc598056c8a5962451271f\": container with ID starting with 6f90fda14f5a6d3496cc77a3dbcb83e40aef661f11fc598056c8a5962451271f not found: ID does not exist" Sep 14 11:29:49.735337 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.735312 2556 scope.go:117] "RemoveContainer" containerID="23abcf0b22754cac9012819f0f6a038e73333c6d383f3c8a841f3ca357ab4933" Sep 14 11:29:49.735548 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:29:49.735529 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23abcf0b22754cac9012819f0f6a038e73333c6d383f3c8a841f3ca357ab4933\": container with ID starting with 23abcf0b22754cac9012819f0f6a038e73333c6d383f3c8a841f3ca357ab4933 not found: ID does not exist" containerID="23abcf0b22754cac9012819f0f6a038e73333c6d383f3c8a841f3ca357ab4933" Sep 14 11:29:49.735601 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.735558 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23abcf0b22754cac9012819f0f6a038e73333c6d383f3c8a841f3ca357ab4933"} err="failed to get container status \"23abcf0b22754cac9012819f0f6a038e73333c6d383f3c8a841f3ca357ab4933\": rpc error: code = NotFound desc = could not find container \"23abcf0b22754cac9012819f0f6a038e73333c6d383f3c8a841f3ca357ab4933\": container with ID starting with 23abcf0b22754cac9012819f0f6a038e73333c6d383f3c8a841f3ca357ab4933 not found: ID does not exist" Sep 14 11:29:49.794531 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.794511 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cq5h2\" (UniqueName: \"kubernetes.io/projected/d7d65916-67c9-42b7-8b55-3311ed30ab2d-kube-api-access-cq5h2\") pod \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " Sep 14 11:29:49.794639 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.794542 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istio-envoy\") pod \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " Sep 14 11:29:49.794639 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.794597 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-credential-socket\") pod \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " Sep 14 11:29:49.794639 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.794615 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istio-podinfo\") pod \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " Sep 14 11:29:49.794810 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.794662 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-workload-certs\") pod \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " Sep 14 11:29:49.794810 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.794707 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istio-token\") pod \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " Sep 14 11:29:49.795191 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.794920 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d7d65916-67c9-42b7-8b55-3311ed30ab2d" (UID: "d7d65916-67c9-42b7-8b55-3311ed30ab2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:49.795191 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.794955 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d7d65916-67c9-42b7-8b55-3311ed30ab2d" (UID: "d7d65916-67c9-42b7-8b55-3311ed30ab2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:49.795191 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.794749 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istio-data\") pod \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " Sep 14 11:29:49.795191 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.795157 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istio-data" (OuterVolumeSpecName: "istio-data") pod "d7d65916-67c9-42b7-8b55-3311ed30ab2d" (UID: "d7d65916-67c9-42b7-8b55-3311ed30ab2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:49.795191 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.795182 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-workload-socket\") pod \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " Sep 14 11:29:49.795541 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.795237 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istiod-ca-cert\") pod \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\" (UID: \"d7d65916-67c9-42b7-8b55-3311ed30ab2d\") " Sep 14 11:29:49.795541 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.795449 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d7d65916-67c9-42b7-8b55-3311ed30ab2d" (UID: "d7d65916-67c9-42b7-8b55-3311ed30ab2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:49.795711 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.795675 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:49.795711 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.795700 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:49.795819 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.795713 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:49.795819 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.795725 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:49.795819 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.795750 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d7d65916-67c9-42b7-8b55-3311ed30ab2d" (UID: "d7d65916-67c9-42b7-8b55-3311ed30ab2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:29:49.796740 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.796713 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d7d65916-67c9-42b7-8b55-3311ed30ab2d" (UID: "d7d65916-67c9-42b7-8b55-3311ed30ab2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:29:49.796825 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.796737 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d7d65916-67c9-42b7-8b55-3311ed30ab2d" (UID: "d7d65916-67c9-42b7-8b55-3311ed30ab2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 11:29:49.796914 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.796888 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d65916-67c9-42b7-8b55-3311ed30ab2d-kube-api-access-cq5h2" (OuterVolumeSpecName: "kube-api-access-cq5h2") pod "d7d65916-67c9-42b7-8b55-3311ed30ab2d" (UID: "d7d65916-67c9-42b7-8b55-3311ed30ab2d"). InnerVolumeSpecName "kube-api-access-cq5h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:29:49.797004 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.796931 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istio-token" (OuterVolumeSpecName: "istio-token") pod "d7d65916-67c9-42b7-8b55-3311ed30ab2d" (UID: "d7d65916-67c9-42b7-8b55-3311ed30ab2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:29:49.896136 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.896114 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:49.896136 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.896134 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:49.896265 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.896143 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq5h2\" (UniqueName: \"kubernetes.io/projected/d7d65916-67c9-42b7-8b55-3311ed30ab2d-kube-api-access-cq5h2\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:49.896265 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.896152 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:49.896265 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:49.896160 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7d65916-67c9-42b7-8b55-3311ed30ab2d-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:29:50.042232 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:50.042209 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst"] Sep 14 11:29:50.050477 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:50.050457 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-420d64-openshift-default-5654854486-b5lst"] Sep 14 11:29:50.383401 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:50.383365 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7d65916-67c9-42b7-8b55-3311ed30ab2d" path="/var/lib/kubelet/pods/d7d65916-67c9-42b7-8b55-3311ed30ab2d/volumes" Sep 14 11:29:54.357979 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:54.357943 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e42fe6-jqpjw"] Sep 14 11:29:54.360121 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:54.360079 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e42fe6-jqpjw"] Sep 14 11:29:54.382463 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:29:54.382440 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e1ecb57-4773-43f6-aecd-a35aa7ebac43" path="/var/lib/kubelet/pods/9e1ecb57-4773-43f6-aecd-a35aa7ebac43/volumes" Sep 14 11:30:00.146307 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.146266 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29823090-6d5x2"] Sep 14 11:30:00.146773 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.146760 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7d65916-67c9-42b7-8b55-3311ed30ab2d" containerName="istio-proxy" Sep 14 11:30:00.146849 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.146780 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7d65916-67c9-42b7-8b55-3311ed30ab2d" containerName="istio-proxy" Sep 14 11:30:00.146849 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.146804 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e1ecb57-4773-43f6-aecd-a35aa7ebac43" containerName="revoke-keys" Sep 14 11:30:00.146849 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.146812 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e1ecb57-4773-43f6-aecd-a35aa7ebac43" containerName="revoke-keys" Sep 14 11:30:00.146849 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.146822 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7d65916-67c9-42b7-8b55-3311ed30ab2d" containerName="istio-proxy" Sep 14 11:30:00.146849 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.146831 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7d65916-67c9-42b7-8b55-3311ed30ab2d" containerName="istio-proxy" Sep 14 11:30:00.147130 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.146922 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e1ecb57-4773-43f6-aecd-a35aa7ebac43" containerName="revoke-keys" Sep 14 11:30:00.147130 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.146933 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7d65916-67c9-42b7-8b55-3311ed30ab2d" containerName="istio-proxy" Sep 14 11:30:00.147130 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.146942 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7d65916-67c9-42b7-8b55-3311ed30ab2d" containerName="istio-proxy" Sep 14 11:30:00.152174 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.152146 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-55c025-29823090-bmw9b"] Sep 14 11:30:00.152479 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.152439 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823090-6d5x2" Sep 14 11:30:00.155694 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.155673 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3f02a5-29823090-p9j4q"] Sep 14 11:30:00.155846 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.155775 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-55c025-29823090-bmw9b" Sep 14 11:30:00.158849 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.158830 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3f02a5-29823090-p9j4q" Sep 14 11:30:00.171593 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.171572 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f98vt\"" Sep 14 11:30:00.176843 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.176814 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29823090-6d5x2"] Sep 14 11:30:00.178255 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.178232 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-55c025-29823090-bmw9b"] Sep 14 11:30:00.178451 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.178424 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823090-6d5x2" Sep 14 11:30:00.179775 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.179754 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3f02a5-29823090-p9j4q"] Sep 14 11:30:00.180768 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.180747 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-55c025-29823090-bmw9b" Sep 14 11:30:00.208130 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.207926 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3f02a5-29823090-p9j4q" Sep 14 11:30:00.480081 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.480048 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29823090-6d5x2"] Sep 14 11:30:00.482017 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:30:00.481990 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f49044a_70b7_4a12_8b02_a45f74ece418.slice/crio-4a1aea88b0db6d05269a627c68ae5ddc338594f6c5fb82434fe292c04f79e950 WatchSource:0}: Error finding container 4a1aea88b0db6d05269a627c68ae5ddc338594f6c5fb82434fe292c04f79e950: Status 404 returned error can't find the container with id 4a1aea88b0db6d05269a627c68ae5ddc338594f6c5fb82434fe292c04f79e950 Sep 14 11:30:00.513753 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.513728 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3f02a5-29823090-p9j4q"] Sep 14 11:30:00.515389 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:30:00.515366 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78120fc_2b21_419f_90b7_e43c3841861b.slice/crio-752887ebc4c6e6d8e10faa5abb920ed4ae01f0d400c2e85daebafdd6bd3b9091 WatchSource:0}: Error finding container 752887ebc4c6e6d8e10faa5abb920ed4ae01f0d400c2e85daebafdd6bd3b9091: Status 404 returned error can't find the container with id 752887ebc4c6e6d8e10faa5abb920ed4ae01f0d400c2e85daebafdd6bd3b9091 Sep 14 11:30:00.553717 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.553683 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-55c025-29823090-bmw9b"] Sep 14 11:30:00.556291 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:30:00.556265 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c1f9a0_070e_4a4d_a9c8_05eaf0381b5f.slice/crio-ca0399d3d2433ec0140875b33bed19e4fc059bba9abf69cd48d9037529002d60 WatchSource:0}: Error finding container ca0399d3d2433ec0140875b33bed19e4fc059bba9abf69cd48d9037529002d60: Status 404 returned error can't find the container with id ca0399d3d2433ec0140875b33bed19e4fc059bba9abf69cd48d9037529002d60 Sep 14 11:30:00.773382 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.773307 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823090-6d5x2" event={"ID":"3f49044a-70b7-4a12-8b02-a45f74ece418","Type":"ContainerStarted","Data":"4a1aea88b0db6d05269a627c68ae5ddc338594f6c5fb82434fe292c04f79e950"} Sep 14 11:30:00.774456 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.774431 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-55c025-29823090-bmw9b" event={"ID":"e5c1f9a0-070e-4a4d-a9c8-05eaf0381b5f","Type":"ContainerStarted","Data":"ca0399d3d2433ec0140875b33bed19e4fc059bba9abf69cd48d9037529002d60"} Sep 14 11:30:00.775911 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.775887 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3f02a5-29823090-p9j4q" event={"ID":"a78120fc-2b21-419f-90b7-e43c3841861b","Type":"ContainerStarted","Data":"5ce39195c1df6630dfe38afdc002f216a28512f52ab22efbaf42e5397a7c851a"} Sep 14 11:30:00.776014 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.775931 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3f02a5-29823090-p9j4q" event={"ID":"a78120fc-2b21-419f-90b7-e43c3841861b","Type":"ContainerStarted","Data":"752887ebc4c6e6d8e10faa5abb920ed4ae01f0d400c2e85daebafdd6bd3b9091"} Sep 14 11:30:00.795075 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:00.794989 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3f02a5-29823090-p9j4q" podStartSLOduration=0.794971902 podStartE2EDuration="794.971902ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:30:00.79342765 +0000 UTC m=+1758.856830563" watchObservedRunningTime="2026-09-14 11:30:00.794971902 +0000 UTC m=+1758.858374812" Sep 14 11:30:01.785837 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:01.785140 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-55c025-29823090-bmw9b" event={"ID":"e5c1f9a0-070e-4a4d-a9c8-05eaf0381b5f","Type":"ContainerStarted","Data":"6ddd9b859084c69c7a9e10b5cadd840781a556fd5c608170096b3099c25f5150"} Sep 14 11:30:01.788282 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:01.788242 2556 generic.go:356] "Generic (PLEG): container finished" podID="a78120fc-2b21-419f-90b7-e43c3841861b" containerID="5ce39195c1df6630dfe38afdc002f216a28512f52ab22efbaf42e5397a7c851a" exitCode=0 Sep 14 11:30:01.788786 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:01.788459 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3f02a5-29823090-p9j4q" event={"ID":"a78120fc-2b21-419f-90b7-e43c3841861b","Type":"ContainerDied","Data":"5ce39195c1df6630dfe38afdc002f216a28512f52ab22efbaf42e5397a7c851a"} Sep 14 11:30:01.790484 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:01.790396 2556 generic.go:356] "Generic (PLEG): container finished" podID="3f49044a-70b7-4a12-8b02-a45f74ece418" containerID="565d40b82ecfa3105bb5ef3cb61733bbd9fa6bf30ecc277c5774e755bd9e9120" exitCode=0 Sep 14 11:30:01.790484 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:01.790476 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823090-6d5x2" event={"ID":"3f49044a-70b7-4a12-8b02-a45f74ece418","Type":"ContainerDied","Data":"565d40b82ecfa3105bb5ef3cb61733bbd9fa6bf30ecc277c5774e755bd9e9120"} Sep 14 11:30:01.791819 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:01.791795 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252"] Sep 14 11:30:01.792117 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:01.792040 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" podUID="1a14228d-a125-4d08-bf6c-f162f74f6330" containerName="istio-proxy" containerID="cri-o://4a7f621a38fe930a7a425016f9be52aa6c83c1342f1218e21d04c3fa15f31caa" gracePeriod=30 Sep 14 11:30:01.803347 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:01.803303 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-55c025-29823090-bmw9b" podStartSLOduration=1.8032887629999999 podStartE2EDuration="1.803288763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:30:01.801994808 +0000 UTC m=+1759.865397724" watchObservedRunningTime="2026-09-14 11:30:01.803288763 +0000 UTC m=+1759.866691672" Sep 14 11:30:02.121496 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:02.121463 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-41ae25-6tzh4"] Sep 14 11:30:02.124680 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:02.124664 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-41ae25-6tzh4" Sep 14 11:30:02.127179 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:02.127158 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 11:30:02.132732 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:02.132706 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-41ae25-6tzh4"] Sep 14 11:30:02.208511 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:02.208451 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe24a37d-c014-4b41-92f0-d6728bd329ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-41ae25-6tzh4\" (UID: \"fe24a37d-c014-4b41-92f0-d6728bd329ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-41ae25-6tzh4" Sep 14 11:30:02.309817 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:02.309791 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe24a37d-c014-4b41-92f0-d6728bd329ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-41ae25-6tzh4\" (UID: \"fe24a37d-c014-4b41-92f0-d6728bd329ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-41ae25-6tzh4" Sep 14 11:30:02.310348 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:02.310330 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe24a37d-c014-4b41-92f0-d6728bd329ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-41ae25-6tzh4\" (UID: \"fe24a37d-c014-4b41-92f0-d6728bd329ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-41ae25-6tzh4" Sep 14 11:30:02.436406 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:02.436374 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-41ae25-6tzh4" Sep 14 11:30:02.708643 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:02.708617 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-41ae25-6tzh4"] Sep 14 11:30:02.710072 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:30:02.710042 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe24a37d_c014_4b41_92f0_d6728bd329ee.slice/crio-45a36459fe570fbd6ebd71b1bccdd429749bf92ae7ef49e47ccd710d1c7795a0 WatchSource:0}: Error finding container 45a36459fe570fbd6ebd71b1bccdd429749bf92ae7ef49e47ccd710d1c7795a0: Status 404 returned error can't find the container with id 45a36459fe570fbd6ebd71b1bccdd429749bf92ae7ef49e47ccd710d1c7795a0 Sep 14 11:30:02.797096 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:02.797056 2556 generic.go:356] "Generic (PLEG): container finished" podID="e5c1f9a0-070e-4a4d-a9c8-05eaf0381b5f" containerID="6ddd9b859084c69c7a9e10b5cadd840781a556fd5c608170096b3099c25f5150" exitCode=0 Sep 14 11:30:02.797400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:02.797144 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-55c025-29823090-bmw9b" event={"ID":"e5c1f9a0-070e-4a4d-a9c8-05eaf0381b5f","Type":"ContainerDied","Data":"6ddd9b859084c69c7a9e10b5cadd840781a556fd5c608170096b3099c25f5150"} Sep 14 11:30:02.798329 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:02.798306 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-41ae25-6tzh4" event={"ID":"fe24a37d-c014-4b41-92f0-d6728bd329ee","Type":"ContainerStarted","Data":"45a36459fe570fbd6ebd71b1bccdd429749bf92ae7ef49e47ccd710d1c7795a0"} Sep 14 11:30:02.927101 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:02.927052 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823090-6d5x2" Sep 14 11:30:02.954497 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:02.954473 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3f02a5-29823090-p9j4q" Sep 14 11:30:03.804745 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:03.804672 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3f02a5-29823090-p9j4q" Sep 14 11:30:03.804745 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:03.804676 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3f02a5-29823090-p9j4q" event={"ID":"a78120fc-2b21-419f-90b7-e43c3841861b","Type":"ContainerDied","Data":"752887ebc4c6e6d8e10faa5abb920ed4ae01f0d400c2e85daebafdd6bd3b9091"} Sep 14 11:30:03.804745 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:03.804726 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="752887ebc4c6e6d8e10faa5abb920ed4ae01f0d400c2e85daebafdd6bd3b9091" Sep 14 11:30:03.806130 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:03.806103 2556 generic.go:356] "Generic (PLEG): container finished" podID="fe24a37d-c014-4b41-92f0-d6728bd329ee" containerID="5e95a351c5cfd850f43f2556f5e76227937d51a9f561979237088814c3193aea" exitCode=0 Sep 14 11:30:03.806239 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:03.806182 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-41ae25-6tzh4" event={"ID":"fe24a37d-c014-4b41-92f0-d6728bd329ee","Type":"ContainerDied","Data":"5e95a351c5cfd850f43f2556f5e76227937d51a9f561979237088814c3193aea"} Sep 14 11:30:03.810640 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:03.810623 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823090-6d5x2" Sep 14 11:30:03.810744 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:03.810655 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823090-6d5x2" event={"ID":"3f49044a-70b7-4a12-8b02-a45f74ece418","Type":"ContainerDied","Data":"4a1aea88b0db6d05269a627c68ae5ddc338594f6c5fb82434fe292c04f79e950"} Sep 14 11:30:03.810744 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:03.810693 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a1aea88b0db6d05269a627c68ae5ddc338594f6c5fb82434fe292c04f79e950" Sep 14 11:30:03.939462 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:03.939441 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-55c025-29823090-bmw9b" Sep 14 11:30:04.233525 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:04.233496 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-de0f7a-lmhv7"] Sep 14 11:30:04.233839 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:04.233827 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5c1f9a0-070e-4a4d-a9c8-05eaf0381b5f" containerName="cleanup" Sep 14 11:30:04.233895 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:04.233840 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5c1f9a0-070e-4a4d-a9c8-05eaf0381b5f" containerName="cleanup" Sep 14 11:30:04.233895 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:04.233857 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a78120fc-2b21-419f-90b7-e43c3841861b" containerName="cleanup" Sep 14 11:30:04.233895 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:04.233862 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a78120fc-2b21-419f-90b7-e43c3841861b" containerName="cleanup" Sep 14 11:30:04.233895 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:04.233880 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f49044a-70b7-4a12-8b02-a45f74ece418" containerName="cleanup" Sep 14 11:30:04.233895 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:04.233886 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f49044a-70b7-4a12-8b02-a45f74ece418" containerName="cleanup" Sep 14 11:30:04.234069 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:04.233939 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f49044a-70b7-4a12-8b02-a45f74ece418" containerName="cleanup" Sep 14 11:30:04.234069 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:04.233948 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5c1f9a0-070e-4a4d-a9c8-05eaf0381b5f" containerName="cleanup" Sep 14 11:30:04.234069 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:04.233953 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a78120fc-2b21-419f-90b7-e43c3841861b" containerName="cleanup" Sep 14 11:30:04.236831 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:04.236815 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-de0f7a-lmhv7" Sep 14 11:30:04.249926 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:04.249904 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-de0f7a-lmhv7"] Sep 14 11:30:04.330540 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:04.330497 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a06fe39-8703-4e7f-bff6-455f29e33f82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-de0f7a-lmhv7\" (UID: \"0a06fe39-8703-4e7f-bff6-455f29e33f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-de0f7a-lmhv7" Sep 14 11:30:04.431785 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:04.431756 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a06fe39-8703-4e7f-bff6-455f29e33f82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-de0f7a-lmhv7\" (UID: \"0a06fe39-8703-4e7f-bff6-455f29e33f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-de0f7a-lmhv7" Sep 14 11:30:04.432380 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:04.432360 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a06fe39-8703-4e7f-bff6-455f29e33f82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-de0f7a-lmhv7\" (UID: \"0a06fe39-8703-4e7f-bff6-455f29e33f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-de0f7a-lmhv7" Sep 14 11:30:04.546069 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:04.546009 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-de0f7a-lmhv7" Sep 14 11:30:04.815348 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:04.815283 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-de0f7a-lmhv7"] Sep 14 11:30:04.817942 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:30:04.817913 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a06fe39_8703_4e7f_bff6_455f29e33f82.slice/crio-5a67fbc3c19d5869c7afcf4e5470d7518e8f17a1017a8886038f0f30cbe6c086 WatchSource:0}: Error finding container 5a67fbc3c19d5869c7afcf4e5470d7518e8f17a1017a8886038f0f30cbe6c086: Status 404 returned error can't find the container with id 5a67fbc3c19d5869c7afcf4e5470d7518e8f17a1017a8886038f0f30cbe6c086 Sep 14 11:30:04.818345 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:04.818322 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-55c025-29823090-bmw9b" Sep 14 11:30:04.818441 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:04.818368 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-55c025-29823090-bmw9b" event={"ID":"e5c1f9a0-070e-4a4d-a9c8-05eaf0381b5f","Type":"ContainerDied","Data":"ca0399d3d2433ec0140875b33bed19e4fc059bba9abf69cd48d9037529002d60"} Sep 14 11:30:04.818441 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:04.818424 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca0399d3d2433ec0140875b33bed19e4fc059bba9abf69cd48d9037529002d60" Sep 14 11:30:04.941181 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:04.941161 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-41ae25-6tzh4" Sep 14 11:30:05.037190 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:05.037165 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe24a37d-c014-4b41-92f0-d6728bd329ee-maas-api-service-ca\") pod \"fe24a37d-c014-4b41-92f0-d6728bd329ee\" (UID: \"fe24a37d-c014-4b41-92f0-d6728bd329ee\") " Sep 14 11:30:05.037474 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:05.037455 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe24a37d-c014-4b41-92f0-d6728bd329ee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fe24a37d-c014-4b41-92f0-d6728bd329ee" (UID: "fe24a37d-c014-4b41-92f0-d6728bd329ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:30:05.138780 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:05.138752 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe24a37d-c014-4b41-92f0-d6728bd329ee-maas-api-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:05.825417 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:05.825339 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-41ae25-6tzh4" Sep 14 11:30:05.825417 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:05.825372 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-41ae25-6tzh4" event={"ID":"fe24a37d-c014-4b41-92f0-d6728bd329ee","Type":"ContainerDied","Data":"45a36459fe570fbd6ebd71b1bccdd429749bf92ae7ef49e47ccd710d1c7795a0"} Sep 14 11:30:05.825825 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:05.825432 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45a36459fe570fbd6ebd71b1bccdd429749bf92ae7ef49e47ccd710d1c7795a0" Sep 14 11:30:05.826946 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:05.826918 2556 generic.go:356] "Generic (PLEG): container finished" podID="0a06fe39-8703-4e7f-bff6-455f29e33f82" containerID="68edfaf0c3af5f81c17ea36c3dfa51a7547f5803c65db12c019473e2967b5559" exitCode=0 Sep 14 11:30:05.827075 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:05.826991 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-de0f7a-lmhv7" event={"ID":"0a06fe39-8703-4e7f-bff6-455f29e33f82","Type":"ContainerDied","Data":"68edfaf0c3af5f81c17ea36c3dfa51a7547f5803c65db12c019473e2967b5559"} Sep 14 11:30:05.827075 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:05.827033 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-de0f7a-lmhv7" event={"ID":"0a06fe39-8703-4e7f-bff6-455f29e33f82","Type":"ContainerStarted","Data":"5a67fbc3c19d5869c7afcf4e5470d7518e8f17a1017a8886038f0f30cbe6c086"} Sep 14 11:30:06.965924 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:06.965895 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-de0f7a-lmhv7" Sep 14 11:30:07.048052 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.048031 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:30:07.056439 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.056386 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a06fe39-8703-4e7f-bff6-455f29e33f82-maas-api-service-ca\") pod \"0a06fe39-8703-4e7f-bff6-455f29e33f82\" (UID: \"0a06fe39-8703-4e7f-bff6-455f29e33f82\") " Sep 14 11:30:07.056439 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.056438 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-credential-socket\") pod \"1a14228d-a125-4d08-bf6c-f162f74f6330\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " Sep 14 11:30:07.056577 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.056457 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-workload-socket\") pod \"1a14228d-a125-4d08-bf6c-f162f74f6330\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " Sep 14 11:30:07.056577 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.056502 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a14228d-a125-4d08-bf6c-f162f74f6330-istiod-ca-cert\") pod \"1a14228d-a125-4d08-bf6c-f162f74f6330\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " Sep 14 11:30:07.056577 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.056530 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s28bb\" (UniqueName: \"kubernetes.io/projected/1a14228d-a125-4d08-bf6c-f162f74f6330-kube-api-access-s28bb\") pod \"1a14228d-a125-4d08-bf6c-f162f74f6330\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " Sep 14 11:30:07.056577 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.056569 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a14228d-a125-4d08-bf6c-f162f74f6330-istio-token\") pod \"1a14228d-a125-4d08-bf6c-f162f74f6330\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " Sep 14 11:30:07.056783 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.056613 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-istio-data\") pod \"1a14228d-a125-4d08-bf6c-f162f74f6330\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " Sep 14 11:30:07.056783 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.056653 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-istio-envoy\") pod \"1a14228d-a125-4d08-bf6c-f162f74f6330\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " Sep 14 11:30:07.056783 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.056710 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a14228d-a125-4d08-bf6c-f162f74f6330-istio-podinfo\") pod \"1a14228d-a125-4d08-bf6c-f162f74f6330\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " Sep 14 11:30:07.056929 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.056779 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1a14228d-a125-4d08-bf6c-f162f74f6330" (UID: "1a14228d-a125-4d08-bf6c-f162f74f6330"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:07.056929 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.056792 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a06fe39-8703-4e7f-bff6-455f29e33f82-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0a06fe39-8703-4e7f-bff6-455f29e33f82" (UID: "0a06fe39-8703-4e7f-bff6-455f29e33f82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:30:07.056929 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.056870 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-istio-data" (OuterVolumeSpecName: "istio-data") pod "1a14228d-a125-4d08-bf6c-f162f74f6330" (UID: "1a14228d-a125-4d08-bf6c-f162f74f6330"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:07.057169 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.056968 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1a14228d-a125-4d08-bf6c-f162f74f6330" (UID: "1a14228d-a125-4d08-bf6c-f162f74f6330"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:07.057169 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.056994 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a14228d-a125-4d08-bf6c-f162f74f6330-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1a14228d-a125-4d08-bf6c-f162f74f6330" (UID: "1a14228d-a125-4d08-bf6c-f162f74f6330"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:30:07.057169 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.057110 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:07.057169 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.057131 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:07.057169 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.057149 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a14228d-a125-4d08-bf6c-f162f74f6330-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:07.057169 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.057166 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:07.057388 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.057177 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a06fe39-8703-4e7f-bff6-455f29e33f82-maas-api-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:07.058587 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.058565 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a14228d-a125-4d08-bf6c-f162f74f6330-kube-api-access-s28bb" (OuterVolumeSpecName: "kube-api-access-s28bb") pod "1a14228d-a125-4d08-bf6c-f162f74f6330" (UID: "1a14228d-a125-4d08-bf6c-f162f74f6330"). InnerVolumeSpecName "kube-api-access-s28bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:30:07.058773 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.058756 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a14228d-a125-4d08-bf6c-f162f74f6330-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1a14228d-a125-4d08-bf6c-f162f74f6330" (UID: "1a14228d-a125-4d08-bf6c-f162f74f6330"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 11:30:07.059059 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.059034 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1a14228d-a125-4d08-bf6c-f162f74f6330" (UID: "1a14228d-a125-4d08-bf6c-f162f74f6330"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:07.059059 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.059048 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a14228d-a125-4d08-bf6c-f162f74f6330-istio-token" (OuterVolumeSpecName: "istio-token") pod "1a14228d-a125-4d08-bf6c-f162f74f6330" (UID: "1a14228d-a125-4d08-bf6c-f162f74f6330"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:30:07.157552 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.157527 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-workload-certs\") pod \"1a14228d-a125-4d08-bf6c-f162f74f6330\" (UID: \"1a14228d-a125-4d08-bf6c-f162f74f6330\") " Sep 14 11:30:07.157705 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.157694 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:07.157751 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.157707 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a14228d-a125-4d08-bf6c-f162f74f6330-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:07.157751 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.157716 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s28bb\" (UniqueName: \"kubernetes.io/projected/1a14228d-a125-4d08-bf6c-f162f74f6330-kube-api-access-s28bb\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:07.157751 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.157724 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a14228d-a125-4d08-bf6c-f162f74f6330-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:07.157842 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.157743 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1a14228d-a125-4d08-bf6c-f162f74f6330" (UID: "1a14228d-a125-4d08-bf6c-f162f74f6330"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:07.258026 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.258003 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a14228d-a125-4d08-bf6c-f162f74f6330-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:07.838694 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.838671 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-de0f7a-lmhv7" Sep 14 11:30:07.838823 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.838664 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-de0f7a-lmhv7" event={"ID":"0a06fe39-8703-4e7f-bff6-455f29e33f82","Type":"ContainerDied","Data":"5a67fbc3c19d5869c7afcf4e5470d7518e8f17a1017a8886038f0f30cbe6c086"} Sep 14 11:30:07.838823 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.838807 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a67fbc3c19d5869c7afcf4e5470d7518e8f17a1017a8886038f0f30cbe6c086" Sep 14 11:30:07.840552 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.840530 2556 generic.go:356] "Generic (PLEG): container finished" podID="1a14228d-a125-4d08-bf6c-f162f74f6330" containerID="4a7f621a38fe930a7a425016f9be52aa6c83c1342f1218e21d04c3fa15f31caa" exitCode=0 Sep 14 11:30:07.840669 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.840557 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" event={"ID":"1a14228d-a125-4d08-bf6c-f162f74f6330","Type":"ContainerDied","Data":"4a7f621a38fe930a7a425016f9be52aa6c83c1342f1218e21d04c3fa15f31caa"} Sep 14 11:30:07.840669 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.840573 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" Sep 14 11:30:07.840669 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.840587 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252" event={"ID":"1a14228d-a125-4d08-bf6c-f162f74f6330","Type":"ContainerDied","Data":"d7e859c169dc55f25ffa074a468cb15f6c52aae0e4db80b528253a98dd4cf86e"} Sep 14 11:30:07.840669 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.840607 2556 scope.go:117] "RemoveContainer" containerID="4a7f621a38fe930a7a425016f9be52aa6c83c1342f1218e21d04c3fa15f31caa" Sep 14 11:30:07.849726 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.849712 2556 scope.go:117] "RemoveContainer" containerID="b09ca90c93278c463ecfc08aabd02bb45c09483cd95720b3795b20ea1ebf594a" Sep 14 11:30:07.857950 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.857935 2556 scope.go:117] "RemoveContainer" containerID="4a7f621a38fe930a7a425016f9be52aa6c83c1342f1218e21d04c3fa15f31caa" Sep 14 11:30:07.858208 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:30:07.858193 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a7f621a38fe930a7a425016f9be52aa6c83c1342f1218e21d04c3fa15f31caa\": container with ID starting with 4a7f621a38fe930a7a425016f9be52aa6c83c1342f1218e21d04c3fa15f31caa not found: ID does not exist" containerID="4a7f621a38fe930a7a425016f9be52aa6c83c1342f1218e21d04c3fa15f31caa" Sep 14 11:30:07.858257 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.858215 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7f621a38fe930a7a425016f9be52aa6c83c1342f1218e21d04c3fa15f31caa"} err="failed to get container status \"4a7f621a38fe930a7a425016f9be52aa6c83c1342f1218e21d04c3fa15f31caa\": rpc error: code = NotFound desc = could not find container \"4a7f621a38fe930a7a425016f9be52aa6c83c1342f1218e21d04c3fa15f31caa\": container with ID starting with 4a7f621a38fe930a7a425016f9be52aa6c83c1342f1218e21d04c3fa15f31caa not found: ID does not exist" Sep 14 11:30:07.858257 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.858230 2556 scope.go:117] "RemoveContainer" containerID="b09ca90c93278c463ecfc08aabd02bb45c09483cd95720b3795b20ea1ebf594a" Sep 14 11:30:07.858458 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:30:07.858441 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b09ca90c93278c463ecfc08aabd02bb45c09483cd95720b3795b20ea1ebf594a\": container with ID starting with b09ca90c93278c463ecfc08aabd02bb45c09483cd95720b3795b20ea1ebf594a not found: ID does not exist" containerID="b09ca90c93278c463ecfc08aabd02bb45c09483cd95720b3795b20ea1ebf594a" Sep 14 11:30:07.858517 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.858468 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b09ca90c93278c463ecfc08aabd02bb45c09483cd95720b3795b20ea1ebf594a"} err="failed to get container status \"b09ca90c93278c463ecfc08aabd02bb45c09483cd95720b3795b20ea1ebf594a\": rpc error: code = NotFound desc = could not find container \"b09ca90c93278c463ecfc08aabd02bb45c09483cd95720b3795b20ea1ebf594a\": container with ID starting with b09ca90c93278c463ecfc08aabd02bb45c09483cd95720b3795b20ea1ebf594a not found: ID does not exist" Sep 14 11:30:07.865433 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.865413 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252"] Sep 14 11:30:07.872285 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:07.872265 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e42fe6-openshift-default-845bd5745d-hm252"] Sep 14 11:30:08.383788 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:08.383758 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a14228d-a125-4d08-bf6c-f162f74f6330" path="/var/lib/kubelet/pods/1a14228d-a125-4d08-bf6c-f162f74f6330/volumes" Sep 14 11:30:09.114685 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:09.114650 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g"] Sep 14 11:30:13.079942 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:13.079912 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-41ae25-6tzh4"] Sep 14 11:30:13.084102 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:13.084056 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-41ae25-6tzh4"] Sep 14 11:30:14.382244 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:14.382202 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe24a37d-c014-4b41-92f0-d6728bd329ee" path="/var/lib/kubelet/pods/fe24a37d-c014-4b41-92f0-d6728bd329ee/volumes" Sep 14 11:30:19.809893 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:19.809865 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj"] Sep 14 11:30:19.810322 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:19.810064 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" podUID="51fd3d77-7af2-4683-8b6b-f6783b654cb4" containerName="istio-proxy" containerID="cri-o://2e447df1dbed19d0b223d6e5f235e66edcd7f7afb8d349beeb726c21c0ae92ee" gracePeriod=30 Sep 14 11:30:21.069283 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:21.069253 2556 patch_prober.go:28] interesting pod/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 14 11:30:21.069753 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:21.069325 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" podUID="51fd3d77-7af2-4683-8b6b-f6783b654cb4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 14 11:30:21.220499 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:21.220458 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-de0f7a-lmhv7"] Sep 14 11:30:21.223230 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:21.223204 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-de0f7a-lmhv7"] Sep 14 11:30:22.381906 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:22.381872 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a06fe39-8703-4e7f-bff6-455f29e33f82" path="/var/lib/kubelet/pods/0a06fe39-8703-4e7f-bff6-455f29e33f82/volumes" Sep 14 11:30:24.133681 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.133642 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" podUID="75ff5d71-327a-4204-973a-2e561b2cf2dc" containerName="main" containerID="cri-o://c0101badae11d15f2186e751b727c4607df0c655a1e01a4cbd885178be3411ca" gracePeriod=45 Sep 14 11:30:24.384115 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.384032 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:30:24.516847 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.516819 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ff5d71-327a-4204-973a-2e561b2cf2dc-model-cache\") pod \"75ff5d71-327a-4204-973a-2e561b2cf2dc\" (UID: \"75ff5d71-327a-4204-973a-2e561b2cf2dc\") " Sep 14 11:30:24.517030 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.516864 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ff5d71-327a-4204-973a-2e561b2cf2dc-tls-certs\") pod \"75ff5d71-327a-4204-973a-2e561b2cf2dc\" (UID: \"75ff5d71-327a-4204-973a-2e561b2cf2dc\") " Sep 14 11:30:24.517030 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.516901 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ff5d71-327a-4204-973a-2e561b2cf2dc-tmp-dir\") pod \"75ff5d71-327a-4204-973a-2e561b2cf2dc\" (UID: \"75ff5d71-327a-4204-973a-2e561b2cf2dc\") " Sep 14 11:30:24.517030 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.516952 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ff5d71-327a-4204-973a-2e561b2cf2dc-dshm\") pod \"75ff5d71-327a-4204-973a-2e561b2cf2dc\" (UID: \"75ff5d71-327a-4204-973a-2e561b2cf2dc\") " Sep 14 11:30:24.517030 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.516999 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75ff5d71-327a-4204-973a-2e561b2cf2dc-home\") pod \"75ff5d71-327a-4204-973a-2e561b2cf2dc\" (UID: \"75ff5d71-327a-4204-973a-2e561b2cf2dc\") " Sep 14 11:30:24.517030 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.517021 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ps88d\" (UniqueName: \"kubernetes.io/projected/75ff5d71-327a-4204-973a-2e561b2cf2dc-kube-api-access-ps88d\") pod \"75ff5d71-327a-4204-973a-2e561b2cf2dc\" (UID: \"75ff5d71-327a-4204-973a-2e561b2cf2dc\") " Sep 14 11:30:24.517327 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.517166 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ff5d71-327a-4204-973a-2e561b2cf2dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "75ff5d71-327a-4204-973a-2e561b2cf2dc" (UID: "75ff5d71-327a-4204-973a-2e561b2cf2dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:24.517327 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.517282 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ff5d71-327a-4204-973a-2e561b2cf2dc-home" (OuterVolumeSpecName: "home") pod "75ff5d71-327a-4204-973a-2e561b2cf2dc" (UID: "75ff5d71-327a-4204-973a-2e561b2cf2dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:24.517536 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.517334 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ff5d71-327a-4204-973a-2e561b2cf2dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "75ff5d71-327a-4204-973a-2e561b2cf2dc" (UID: "75ff5d71-327a-4204-973a-2e561b2cf2dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:24.517757 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.517735 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ff5d71-327a-4204-973a-2e561b2cf2dc-tmp-dir\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:24.517924 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.517901 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75ff5d71-327a-4204-973a-2e561b2cf2dc-home\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:24.518030 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.517927 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ff5d71-327a-4204-973a-2e561b2cf2dc-model-cache\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:24.519294 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.519270 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75ff5d71-327a-4204-973a-2e561b2cf2dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75ff5d71-327a-4204-973a-2e561b2cf2dc" (UID: "75ff5d71-327a-4204-973a-2e561b2cf2dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:30:24.519294 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.519281 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ff5d71-327a-4204-973a-2e561b2cf2dc-dshm" (OuterVolumeSpecName: "dshm") pod "75ff5d71-327a-4204-973a-2e561b2cf2dc" (UID: "75ff5d71-327a-4204-973a-2e561b2cf2dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:24.519428 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.519338 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ff5d71-327a-4204-973a-2e561b2cf2dc-kube-api-access-ps88d" (OuterVolumeSpecName: "kube-api-access-ps88d") pod "75ff5d71-327a-4204-973a-2e561b2cf2dc" (UID: "75ff5d71-327a-4204-973a-2e561b2cf2dc"). InnerVolumeSpecName "kube-api-access-ps88d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:30:24.619179 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.619147 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ff5d71-327a-4204-973a-2e561b2cf2dc-tls-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:24.619179 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.619173 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ff5d71-327a-4204-973a-2e561b2cf2dc-dshm\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:24.619179 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.619182 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps88d\" (UniqueName: \"kubernetes.io/projected/75ff5d71-327a-4204-973a-2e561b2cf2dc-kube-api-access-ps88d\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:24.933168 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.933134 2556 generic.go:356] "Generic (PLEG): container finished" podID="51fd3d77-7af2-4683-8b6b-f6783b654cb4" containerID="2e447df1dbed19d0b223d6e5f235e66edcd7f7afb8d349beeb726c21c0ae92ee" exitCode=0 Sep 14 11:30:24.933292 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.933205 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" event={"ID":"51fd3d77-7af2-4683-8b6b-f6783b654cb4","Type":"ContainerDied","Data":"2e447df1dbed19d0b223d6e5f235e66edcd7f7afb8d349beeb726c21c0ae92ee"} Sep 14 11:30:24.933292 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.933260 2556 scope.go:117] "RemoveContainer" containerID="002c204f76ea40de1a13d2f46433527665a4dbbe4ddd811b3fd87fce6b2aea11" Sep 14 11:30:24.934630 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.934603 2556 generic.go:356] "Generic (PLEG): container finished" podID="75ff5d71-327a-4204-973a-2e561b2cf2dc" containerID="c0101badae11d15f2186e751b727c4607df0c655a1e01a4cbd885178be3411ca" exitCode=0 Sep 14 11:30:24.934719 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.934642 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" Sep 14 11:30:24.934719 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.934670 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" event={"ID":"75ff5d71-327a-4204-973a-2e561b2cf2dc","Type":"ContainerDied","Data":"c0101badae11d15f2186e751b727c4607df0c655a1e01a4cbd885178be3411ca"} Sep 14 11:30:24.934719 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.934700 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g" event={"ID":"75ff5d71-327a-4204-973a-2e561b2cf2dc","Type":"ContainerDied","Data":"a8d1591894065db9419ddd45cd856c2496539c845fffecd3784e280e83d5be6b"} Sep 14 11:30:24.944473 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.944454 2556 scope.go:117] "RemoveContainer" containerID="c0101badae11d15f2186e751b727c4607df0c655a1e01a4cbd885178be3411ca" Sep 14 11:30:24.952199 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.952177 2556 scope.go:117] "RemoveContainer" containerID="c0101badae11d15f2186e751b727c4607df0c655a1e01a4cbd885178be3411ca" Sep 14 11:30:24.952451 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:30:24.952433 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0101badae11d15f2186e751b727c4607df0c655a1e01a4cbd885178be3411ca\": container with ID starting with c0101badae11d15f2186e751b727c4607df0c655a1e01a4cbd885178be3411ca not found: ID does not exist" containerID="c0101badae11d15f2186e751b727c4607df0c655a1e01a4cbd885178be3411ca" Sep 14 11:30:24.952522 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.952463 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0101badae11d15f2186e751b727c4607df0c655a1e01a4cbd885178be3411ca"} err="failed to get container status \"c0101badae11d15f2186e751b727c4607df0c655a1e01a4cbd885178be3411ca\": rpc error: code = NotFound desc = could not find container \"c0101badae11d15f2186e751b727c4607df0c655a1e01a4cbd885178be3411ca\": container with ID starting with c0101badae11d15f2186e751b727c4607df0c655a1e01a4cbd885178be3411ca not found: ID does not exist" Sep 14 11:30:24.957140 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.957118 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g"] Sep 14 11:30:24.961903 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:24.961884 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/distinct-2-w4-de0f7a-kserve-c4474fd4f-z8r8g"] Sep 14 11:30:25.056331 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.056310 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:30:25.122145 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.122118 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgsqg\" (UniqueName: \"kubernetes.io/projected/51fd3d77-7af2-4683-8b6b-f6783b654cb4-kube-api-access-mgsqg\") pod \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " Sep 14 11:30:25.122276 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.122156 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istio-envoy\") pod \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " Sep 14 11:30:25.122276 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.122181 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istio-token\") pod \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " Sep 14 11:30:25.122276 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.122213 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istio-podinfo\") pod \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " Sep 14 11:30:25.122276 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.122263 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-workload-certs\") pod \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " Sep 14 11:30:25.122496 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.122287 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istio-data\") pod \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " Sep 14 11:30:25.122496 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.122352 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-credential-socket\") pod \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " Sep 14 11:30:25.122496 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.122379 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-workload-socket\") pod \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " Sep 14 11:30:25.122496 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.122423 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istiod-ca-cert\") pod \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\" (UID: \"51fd3d77-7af2-4683-8b6b-f6783b654cb4\") " Sep 14 11:30:25.122686 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.122560 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "51fd3d77-7af2-4683-8b6b-f6783b654cb4" (UID: "51fd3d77-7af2-4683-8b6b-f6783b654cb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:25.122747 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.122725 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istio-data" (OuterVolumeSpecName: "istio-data") pod "51fd3d77-7af2-4683-8b6b-f6783b654cb4" (UID: "51fd3d77-7af2-4683-8b6b-f6783b654cb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:25.122805 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.122772 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:25.122913 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.122893 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "51fd3d77-7af2-4683-8b6b-f6783b654cb4" (UID: "51fd3d77-7af2-4683-8b6b-f6783b654cb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:30:25.123008 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.122927 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "51fd3d77-7af2-4683-8b6b-f6783b654cb4" (UID: "51fd3d77-7af2-4683-8b6b-f6783b654cb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:25.123076 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.123050 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "51fd3d77-7af2-4683-8b6b-f6783b654cb4" (UID: "51fd3d77-7af2-4683-8b6b-f6783b654cb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:25.124169 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.124150 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "51fd3d77-7af2-4683-8b6b-f6783b654cb4" (UID: "51fd3d77-7af2-4683-8b6b-f6783b654cb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:25.124286 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.124264 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "51fd3d77-7af2-4683-8b6b-f6783b654cb4" (UID: "51fd3d77-7af2-4683-8b6b-f6783b654cb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 11:30:25.124565 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.124547 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51fd3d77-7af2-4683-8b6b-f6783b654cb4-kube-api-access-mgsqg" (OuterVolumeSpecName: "kube-api-access-mgsqg") pod "51fd3d77-7af2-4683-8b6b-f6783b654cb4" (UID: "51fd3d77-7af2-4683-8b6b-f6783b654cb4"). InnerVolumeSpecName "kube-api-access-mgsqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:30:25.124565 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.124558 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istio-token" (OuterVolumeSpecName: "istio-token") pod "51fd3d77-7af2-4683-8b6b-f6783b654cb4" (UID: "51fd3d77-7af2-4683-8b6b-f6783b654cb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:30:25.223366 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.223345 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:25.223644 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.223367 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:25.223644 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.223380 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:25.223644 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.223392 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:25.223644 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.223407 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgsqg\" (UniqueName: \"kubernetes.io/projected/51fd3d77-7af2-4683-8b6b-f6783b654cb4-kube-api-access-mgsqg\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:25.223644 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.223420 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:25.223644 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.223433 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:25.223644 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.223446 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fd3d77-7af2-4683-8b6b-f6783b654cb4-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:25.942135 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.942105 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" Sep 14 11:30:25.942302 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.942148 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj" event={"ID":"51fd3d77-7af2-4683-8b6b-f6783b654cb4","Type":"ContainerDied","Data":"622a55dd3dd665c5eafb104b7cbaa8901f55042dde103296fa6a94aff24d8259"} Sep 14 11:30:25.942302 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.942203 2556 scope.go:117] "RemoveContainer" containerID="2e447df1dbed19d0b223d6e5f235e66edcd7f7afb8d349beeb726c21c0ae92ee" Sep 14 11:30:25.964836 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.964808 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj"] Sep 14 11:30:25.967880 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:25.967860 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-41ae25-openshift-default-6db7959784-hhkmj"] Sep 14 11:30:26.381428 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:26.381391 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51fd3d77-7af2-4683-8b6b-f6783b654cb4" path="/var/lib/kubelet/pods/51fd3d77-7af2-4683-8b6b-f6783b654cb4/volumes" Sep 14 11:30:26.381941 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:26.381919 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75ff5d71-327a-4204-973a-2e561b2cf2dc" path="/var/lib/kubelet/pods/75ff5d71-327a-4204-973a-2e561b2cf2dc/volumes" Sep 14 11:30:28.304630 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:28.304595 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt"] Sep 14 11:30:28.305067 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:28.304886 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" podUID="e39b3e46-7504-46f1-8c15-9ad48b95123b" containerName="istio-proxy" containerID="cri-o://8eaec34685ecabe86146f32a2c3266ea0c60b96ab38fe5b951834972840c4d2e" gracePeriod=30 Sep 14 11:30:33.548573 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.548549 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:30:33.692075 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.692037 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqp99\" (UniqueName: \"kubernetes.io/projected/e39b3e46-7504-46f1-8c15-9ad48b95123b-kube-api-access-qqp99\") pod \"e39b3e46-7504-46f1-8c15-9ad48b95123b\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " Sep 14 11:30:33.692277 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.692110 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39b3e46-7504-46f1-8c15-9ad48b95123b-istio-token\") pod \"e39b3e46-7504-46f1-8c15-9ad48b95123b\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " Sep 14 11:30:33.692277 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.692135 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-credential-socket\") pod \"e39b3e46-7504-46f1-8c15-9ad48b95123b\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " Sep 14 11:30:33.692277 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.692160 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39b3e46-7504-46f1-8c15-9ad48b95123b-istiod-ca-cert\") pod \"e39b3e46-7504-46f1-8c15-9ad48b95123b\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " Sep 14 11:30:33.692446 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.692301 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-workload-socket\") pod \"e39b3e46-7504-46f1-8c15-9ad48b95123b\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " Sep 14 11:30:33.692446 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.692362 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-istio-data\") pod \"e39b3e46-7504-46f1-8c15-9ad48b95123b\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " Sep 14 11:30:33.692446 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.692398 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39b3e46-7504-46f1-8c15-9ad48b95123b-istio-podinfo\") pod \"e39b3e46-7504-46f1-8c15-9ad48b95123b\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " Sep 14 11:30:33.692446 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.692428 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-workload-certs\") pod \"e39b3e46-7504-46f1-8c15-9ad48b95123b\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " Sep 14 11:30:33.692659 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.692472 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e39b3e46-7504-46f1-8c15-9ad48b95123b" (UID: "e39b3e46-7504-46f1-8c15-9ad48b95123b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:33.692659 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.692481 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-istio-envoy\") pod \"e39b3e46-7504-46f1-8c15-9ad48b95123b\" (UID: \"e39b3e46-7504-46f1-8c15-9ad48b95123b\") " Sep 14 11:30:33.692659 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.692570 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e39b3e46-7504-46f1-8c15-9ad48b95123b" (UID: "e39b3e46-7504-46f1-8c15-9ad48b95123b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:33.692659 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.692580 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e39b3e46-7504-46f1-8c15-9ad48b95123b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e39b3e46-7504-46f1-8c15-9ad48b95123b" (UID: "e39b3e46-7504-46f1-8c15-9ad48b95123b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:30:33.692863 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.692786 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-istio-data" (OuterVolumeSpecName: "istio-data") pod "e39b3e46-7504-46f1-8c15-9ad48b95123b" (UID: "e39b3e46-7504-46f1-8c15-9ad48b95123b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:33.692863 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.692836 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:33.692863 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.692856 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:33.693022 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.692873 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39b3e46-7504-46f1-8c15-9ad48b95123b-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:33.693022 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.692939 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e39b3e46-7504-46f1-8c15-9ad48b95123b" (UID: "e39b3e46-7504-46f1-8c15-9ad48b95123b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:33.694739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.694705 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39b3e46-7504-46f1-8c15-9ad48b95123b-kube-api-access-qqp99" (OuterVolumeSpecName: "kube-api-access-qqp99") pod "e39b3e46-7504-46f1-8c15-9ad48b95123b" (UID: "e39b3e46-7504-46f1-8c15-9ad48b95123b"). InnerVolumeSpecName "kube-api-access-qqp99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:30:33.694950 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.694743 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e39b3e46-7504-46f1-8c15-9ad48b95123b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e39b3e46-7504-46f1-8c15-9ad48b95123b" (UID: "e39b3e46-7504-46f1-8c15-9ad48b95123b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 11:30:33.694950 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.694758 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39b3e46-7504-46f1-8c15-9ad48b95123b-istio-token" (OuterVolumeSpecName: "istio-token") pod "e39b3e46-7504-46f1-8c15-9ad48b95123b" (UID: "e39b3e46-7504-46f1-8c15-9ad48b95123b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:30:33.694950 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.694842 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e39b3e46-7504-46f1-8c15-9ad48b95123b" (UID: "e39b3e46-7504-46f1-8c15-9ad48b95123b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:33.794316 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.794286 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:33.794316 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.794310 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39b3e46-7504-46f1-8c15-9ad48b95123b-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:33.794316 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.794318 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:33.794523 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.794326 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39b3e46-7504-46f1-8c15-9ad48b95123b-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:33.794523 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.794334 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqp99\" (UniqueName: \"kubernetes.io/projected/e39b3e46-7504-46f1-8c15-9ad48b95123b-kube-api-access-qqp99\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:33.794523 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.794342 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39b3e46-7504-46f1-8c15-9ad48b95123b-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:33.986311 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.986218 2556 generic.go:356] "Generic (PLEG): container finished" podID="e39b3e46-7504-46f1-8c15-9ad48b95123b" containerID="8eaec34685ecabe86146f32a2c3266ea0c60b96ab38fe5b951834972840c4d2e" exitCode=0 Sep 14 11:30:33.986311 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.986284 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" Sep 14 11:30:33.986534 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.986307 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" event={"ID":"e39b3e46-7504-46f1-8c15-9ad48b95123b","Type":"ContainerDied","Data":"8eaec34685ecabe86146f32a2c3266ea0c60b96ab38fe5b951834972840c4d2e"} Sep 14 11:30:33.986534 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.986367 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt" event={"ID":"e39b3e46-7504-46f1-8c15-9ad48b95123b","Type":"ContainerDied","Data":"7a7af31112ce032b70436a2a0fff1a7d1e41cf19b9c69d8e9a348cee14a8fc17"} Sep 14 11:30:33.986534 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.986394 2556 scope.go:117] "RemoveContainer" containerID="8eaec34685ecabe86146f32a2c3266ea0c60b96ab38fe5b951834972840c4d2e" Sep 14 11:30:33.995384 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:33.995368 2556 scope.go:117] "RemoveContainer" containerID="dd46bac4364093ca5a2c7e23d6a097cba6df4c9c227ef85f3d8eb2c317336a2a" Sep 14 11:30:34.003482 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:34.003463 2556 scope.go:117] "RemoveContainer" containerID="8eaec34685ecabe86146f32a2c3266ea0c60b96ab38fe5b951834972840c4d2e" Sep 14 11:30:34.003759 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:30:34.003735 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eaec34685ecabe86146f32a2c3266ea0c60b96ab38fe5b951834972840c4d2e\": container with ID starting with 8eaec34685ecabe86146f32a2c3266ea0c60b96ab38fe5b951834972840c4d2e not found: ID does not exist" containerID="8eaec34685ecabe86146f32a2c3266ea0c60b96ab38fe5b951834972840c4d2e" Sep 14 11:30:34.003816 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:34.003772 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eaec34685ecabe86146f32a2c3266ea0c60b96ab38fe5b951834972840c4d2e"} err="failed to get container status \"8eaec34685ecabe86146f32a2c3266ea0c60b96ab38fe5b951834972840c4d2e\": rpc error: code = NotFound desc = could not find container \"8eaec34685ecabe86146f32a2c3266ea0c60b96ab38fe5b951834972840c4d2e\": container with ID starting with 8eaec34685ecabe86146f32a2c3266ea0c60b96ab38fe5b951834972840c4d2e not found: ID does not exist" Sep 14 11:30:34.003816 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:34.003796 2556 scope.go:117] "RemoveContainer" containerID="dd46bac4364093ca5a2c7e23d6a097cba6df4c9c227ef85f3d8eb2c317336a2a" Sep 14 11:30:34.004042 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:30:34.004026 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd46bac4364093ca5a2c7e23d6a097cba6df4c9c227ef85f3d8eb2c317336a2a\": container with ID starting with dd46bac4364093ca5a2c7e23d6a097cba6df4c9c227ef85f3d8eb2c317336a2a not found: ID does not exist" containerID="dd46bac4364093ca5a2c7e23d6a097cba6df4c9c227ef85f3d8eb2c317336a2a" Sep 14 11:30:34.004115 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:34.004048 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd46bac4364093ca5a2c7e23d6a097cba6df4c9c227ef85f3d8eb2c317336a2a"} err="failed to get container status \"dd46bac4364093ca5a2c7e23d6a097cba6df4c9c227ef85f3d8eb2c317336a2a\": rpc error: code = NotFound desc = could not find container \"dd46bac4364093ca5a2c7e23d6a097cba6df4c9c227ef85f3d8eb2c317336a2a\": container with ID starting with dd46bac4364093ca5a2c7e23d6a097cba6df4c9c227ef85f3d8eb2c317336a2a not found: ID does not exist" Sep 14 11:30:34.014721 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:34.014700 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt"] Sep 14 11:30:34.022018 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:34.021996 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-de0f7a-openshift-default-8445bc7966-64qdt"] Sep 14 11:30:34.382911 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:34.382879 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e39b3e46-7504-46f1-8c15-9ad48b95123b" path="/var/lib/kubelet/pods/e39b3e46-7504-46f1-8c15-9ad48b95123b/volumes" Sep 14 11:30:51.429390 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429355 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7938-b4ec68"] Sep 14 11:30:51.429784 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429670 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a14228d-a125-4d08-bf6c-f162f74f6330" containerName="istio-proxy" Sep 14 11:30:51.429784 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429682 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a14228d-a125-4d08-bf6c-f162f74f6330" containerName="istio-proxy" Sep 14 11:30:51.429784 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429689 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51fd3d77-7af2-4683-8b6b-f6783b654cb4" containerName="istio-proxy" Sep 14 11:30:51.429784 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429695 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="51fd3d77-7af2-4683-8b6b-f6783b654cb4" containerName="istio-proxy" Sep 14 11:30:51.429784 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429705 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a14228d-a125-4d08-bf6c-f162f74f6330" containerName="istio-proxy" Sep 14 11:30:51.429784 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429713 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a14228d-a125-4d08-bf6c-f162f74f6330" containerName="istio-proxy" Sep 14 11:30:51.429784 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429724 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51fd3d77-7af2-4683-8b6b-f6783b654cb4" containerName="istio-proxy" Sep 14 11:30:51.429784 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429729 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="51fd3d77-7af2-4683-8b6b-f6783b654cb4" containerName="istio-proxy" Sep 14 11:30:51.429784 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429736 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e39b3e46-7504-46f1-8c15-9ad48b95123b" containerName="istio-proxy" Sep 14 11:30:51.429784 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429741 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e39b3e46-7504-46f1-8c15-9ad48b95123b" containerName="istio-proxy" Sep 14 11:30:51.429784 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429749 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a06fe39-8703-4e7f-bff6-455f29e33f82" containerName="revoke-keys" Sep 14 11:30:51.429784 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429753 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a06fe39-8703-4e7f-bff6-455f29e33f82" containerName="revoke-keys" Sep 14 11:30:51.429784 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429764 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75ff5d71-327a-4204-973a-2e561b2cf2dc" containerName="main" Sep 14 11:30:51.429784 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429769 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="75ff5d71-327a-4204-973a-2e561b2cf2dc" containerName="main" Sep 14 11:30:51.429784 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429780 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe24a37d-c014-4b41-92f0-d6728bd329ee" containerName="revoke-keys" Sep 14 11:30:51.429784 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429789 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe24a37d-c014-4b41-92f0-d6728bd329ee" containerName="revoke-keys" Sep 14 11:30:51.430321 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429836 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a14228d-a125-4d08-bf6c-f162f74f6330" containerName="istio-proxy" Sep 14 11:30:51.430321 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429844 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a14228d-a125-4d08-bf6c-f162f74f6330" containerName="istio-proxy" Sep 14 11:30:51.430321 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429850 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e39b3e46-7504-46f1-8c15-9ad48b95123b" containerName="istio-proxy" Sep 14 11:30:51.430321 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429858 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="75ff5d71-327a-4204-973a-2e561b2cf2dc" containerName="main" Sep 14 11:30:51.430321 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429867 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="51fd3d77-7af2-4683-8b6b-f6783b654cb4" containerName="istio-proxy" Sep 14 11:30:51.430321 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429873 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e39b3e46-7504-46f1-8c15-9ad48b95123b" containerName="istio-proxy" Sep 14 11:30:51.430321 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429878 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="51fd3d77-7af2-4683-8b6b-f6783b654cb4" containerName="istio-proxy" Sep 14 11:30:51.430321 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429884 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a06fe39-8703-4e7f-bff6-455f29e33f82" containerName="revoke-keys" Sep 14 11:30:51.430321 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.429889 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe24a37d-c014-4b41-92f0-d6728bd329ee" containerName="revoke-keys" Sep 14 11:30:51.432703 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.432686 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-b4ec68" Sep 14 11:30:51.435866 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.435841 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5kv55\"" Sep 14 11:30:51.437914 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.437890 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7938-b4ec68"] Sep 14 11:30:51.537700 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.537666 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mh7t\" (UniqueName: \"kubernetes.io/projected/782ec688-7a87-4251-bd4e-5d050e28f6b1-kube-api-access-8mh7t\") pod \"test-curl-7938-b4ec68\" (UID: \"782ec688-7a87-4251-bd4e-5d050e28f6b1\") " pod="opendatahub/test-curl-7938-b4ec68" Sep 14 11:30:51.638752 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.638722 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mh7t\" (UniqueName: \"kubernetes.io/projected/782ec688-7a87-4251-bd4e-5d050e28f6b1-kube-api-access-8mh7t\") pod \"test-curl-7938-b4ec68\" (UID: \"782ec688-7a87-4251-bd4e-5d050e28f6b1\") " pod="opendatahub/test-curl-7938-b4ec68" Sep 14 11:30:51.646737 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.646709 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mh7t\" (UniqueName: \"kubernetes.io/projected/782ec688-7a87-4251-bd4e-5d050e28f6b1-kube-api-access-8mh7t\") pod \"test-curl-7938-b4ec68\" (UID: \"782ec688-7a87-4251-bd4e-5d050e28f6b1\") " pod="opendatahub/test-curl-7938-b4ec68" Sep 14 11:30:51.742543 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:51.742483 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-b4ec68" Sep 14 11:30:52.008187 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:52.008162 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7938-b4ec68"] Sep 14 11:30:52.009766 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:30:52.009738 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod782ec688_7a87_4251_bd4e_5d050e28f6b1.slice/crio-d04002d2893421866a51d4c5c4ba44e0785af8848f24d6d3e36e0623d2d5d433 WatchSource:0}: Error finding container d04002d2893421866a51d4c5c4ba44e0785af8848f24d6d3e36e0623d2d5d433: Status 404 returned error can't find the container with id d04002d2893421866a51d4c5c4ba44e0785af8848f24d6d3e36e0623d2d5d433 Sep 14 11:30:52.077684 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:52.077653 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7938-b4ec68" event={"ID":"782ec688-7a87-4251-bd4e-5d050e28f6b1","Type":"ContainerStarted","Data":"d04002d2893421866a51d4c5c4ba44e0785af8848f24d6d3e36e0623d2d5d433"} Sep 14 11:30:53.084401 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:53.084367 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7938-b4ec68" event={"ID":"782ec688-7a87-4251-bd4e-5d050e28f6b1","Type":"ContainerStarted","Data":"09e99b3e3aace86fdad0875449e9813fde98f12287a44a5f9244afa59cfbaaf8"} Sep 14 11:30:53.669742 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:53.669691 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7938-b4ec68" podStartSLOduration=2.66967571 podStartE2EDuration="2.66967571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:30:53.099692347 +0000 UTC m=+1811.163095257" watchObservedRunningTime="2026-09-14 11:30:53.66967571 +0000 UTC m=+1811.733078661" Sep 14 11:30:53.671180 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:53.671161 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7938-b4ec68"] Sep 14 11:30:53.675285 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:53.675262 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7938-b4ec68"] Sep 14 11:30:54.089055 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:30:54.089025 2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7938-b4ec68" container="test-curl-7938-b4ec68" Sep 14 11:30:55.093376 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:55.093326 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7938-b4ec68" podUID="782ec688-7a87-4251-bd4e-5d050e28f6b1" containerName="test-curl-7938-b4ec68" containerID="cri-o://09e99b3e3aace86fdad0875449e9813fde98f12287a44a5f9244afa59cfbaaf8" gracePeriod=2 Sep 14 11:30:57.254027 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:57.253997 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-b4ec68" Sep 14 11:30:57.385053 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:57.385027 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mh7t\" (UniqueName: \"kubernetes.io/projected/782ec688-7a87-4251-bd4e-5d050e28f6b1-kube-api-access-8mh7t\") pod \"782ec688-7a87-4251-bd4e-5d050e28f6b1\" (UID: \"782ec688-7a87-4251-bd4e-5d050e28f6b1\") " Sep 14 11:30:57.386923 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:57.386899 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/782ec688-7a87-4251-bd4e-5d050e28f6b1-kube-api-access-8mh7t" (OuterVolumeSpecName: "kube-api-access-8mh7t") pod "782ec688-7a87-4251-bd4e-5d050e28f6b1" (UID: "782ec688-7a87-4251-bd4e-5d050e28f6b1"). InnerVolumeSpecName "kube-api-access-8mh7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:30:57.486412 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:57.486383 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mh7t\" (UniqueName: \"kubernetes.io/projected/782ec688-7a87-4251-bd4e-5d050e28f6b1-kube-api-access-8mh7t\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:30:58.108445 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:58.108406 2556 generic.go:356] "Generic (PLEG): container finished" podID="782ec688-7a87-4251-bd4e-5d050e28f6b1" containerID="09e99b3e3aace86fdad0875449e9813fde98f12287a44a5f9244afa59cfbaaf8" exitCode=137 Sep 14 11:30:58.108445 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:58.108446 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-b4ec68" Sep 14 11:30:58.108685 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:58.108458 2556 scope.go:117] "RemoveContainer" containerID="09e99b3e3aace86fdad0875449e9813fde98f12287a44a5f9244afa59cfbaaf8" Sep 14 11:30:58.117581 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:58.117568 2556 scope.go:117] "RemoveContainer" containerID="09e99b3e3aace86fdad0875449e9813fde98f12287a44a5f9244afa59cfbaaf8" Sep 14 11:30:58.117839 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:30:58.117821 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09e99b3e3aace86fdad0875449e9813fde98f12287a44a5f9244afa59cfbaaf8\": container with ID starting with 09e99b3e3aace86fdad0875449e9813fde98f12287a44a5f9244afa59cfbaaf8 not found: ID does not exist" containerID="09e99b3e3aace86fdad0875449e9813fde98f12287a44a5f9244afa59cfbaaf8" Sep 14 11:30:58.117903 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:58.117854 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e99b3e3aace86fdad0875449e9813fde98f12287a44a5f9244afa59cfbaaf8"} err="failed to get container status \"09e99b3e3aace86fdad0875449e9813fde98f12287a44a5f9244afa59cfbaaf8\": rpc error: code = NotFound desc = could not find container \"09e99b3e3aace86fdad0875449e9813fde98f12287a44a5f9244afa59cfbaaf8\": container with ID starting with 09e99b3e3aace86fdad0875449e9813fde98f12287a44a5f9244afa59cfbaaf8 not found: ID does not exist" Sep 14 11:30:58.118616 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:30:58.118595 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7938-b4ec68\" is forbidden: User \"system:node:ip-10-0-143-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-52.ec2.internal' and this object" podUID="782ec688-7a87-4251-bd4e-5d050e28f6b1" pod="opendatahub/test-curl-7938-b4ec68" Sep 14 11:30:58.381631 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:30:58.381578 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="782ec688-7a87-4251-bd4e-5d050e28f6b1" path="/var/lib/kubelet/pods/782ec688-7a87-4251-bd4e-5d050e28f6b1/volumes" Sep 14 11:31:08.559109 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.557107 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p"] Sep 14 11:31:08.559109 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.557831 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="782ec688-7a87-4251-bd4e-5d050e28f6b1" containerName="test-curl-7938-b4ec68" Sep 14 11:31:08.559109 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.557853 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="782ec688-7a87-4251-bd4e-5d050e28f6b1" containerName="test-curl-7938-b4ec68" Sep 14 11:31:08.559109 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.557881 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e39b3e46-7504-46f1-8c15-9ad48b95123b" containerName="istio-proxy" Sep 14 11:31:08.559109 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.557890 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e39b3e46-7504-46f1-8c15-9ad48b95123b" containerName="istio-proxy" Sep 14 11:31:08.559109 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.558021 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="782ec688-7a87-4251-bd4e-5d050e28f6b1" containerName="test-curl-7938-b4ec68" Sep 14 11:31:08.562933 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.562907 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:31:08.564242 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.564211 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75f58274-39a8-4c78-85bc-cbd464e896b5-dshm\") pod \"sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p\" (UID: \"75f58274-39a8-4c78-85bc-cbd464e896b5\") " pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:31:08.564369 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.564314 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75f58274-39a8-4c78-85bc-cbd464e896b5-tls-certs\") pod \"sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p\" (UID: \"75f58274-39a8-4c78-85bc-cbd464e896b5\") " pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:31:08.564369 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.564348 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75f58274-39a8-4c78-85bc-cbd464e896b5-home\") pod \"sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p\" (UID: \"75f58274-39a8-4c78-85bc-cbd464e896b5\") " pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:31:08.564497 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.564370 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75f58274-39a8-4c78-85bc-cbd464e896b5-model-cache\") pod \"sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p\" (UID: \"75f58274-39a8-4c78-85bc-cbd464e896b5\") " pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:31:08.564497 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.564418 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgdkg\" (UniqueName: \"kubernetes.io/projected/75f58274-39a8-4c78-85bc-cbd464e896b5-kube-api-access-tgdkg\") pod \"sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p\" (UID: \"75f58274-39a8-4c78-85bc-cbd464e896b5\") " pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:31:08.564497 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.564448 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75f58274-39a8-4c78-85bc-cbd464e896b5-tmp-dir\") pod \"sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p\" (UID: \"75f58274-39a8-4c78-85bc-cbd464e896b5\") " pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:31:08.566134 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.566113 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-55c025\"/\"default-dockercfg-tfffw\"" Sep 14 11:31:08.566134 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.566131 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-55c025\"/\"openshift-service-ca.crt\"" Sep 14 11:31:08.567356 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.567336 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-55c025\"/\"sub-test-model-55c025-kserve-self-signed-certs\"" Sep 14 11:31:08.567420 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.567336 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-55c025\"/\"kube-root-ca.crt\"" Sep 14 11:31:08.570710 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.570691 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p"] Sep 14 11:31:08.664912 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.664884 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75f58274-39a8-4c78-85bc-cbd464e896b5-dshm\") pod \"sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p\" (UID: \"75f58274-39a8-4c78-85bc-cbd464e896b5\") " pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:31:08.665063 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.664943 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75f58274-39a8-4c78-85bc-cbd464e896b5-tls-certs\") pod \"sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p\" (UID: \"75f58274-39a8-4c78-85bc-cbd464e896b5\") " pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:31:08.665063 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.665050 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75f58274-39a8-4c78-85bc-cbd464e896b5-home\") pod \"sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p\" (UID: \"75f58274-39a8-4c78-85bc-cbd464e896b5\") " pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:31:08.665217 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.665077 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75f58274-39a8-4c78-85bc-cbd464e896b5-model-cache\") pod \"sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p\" (UID: \"75f58274-39a8-4c78-85bc-cbd464e896b5\") " pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:31:08.665217 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.665139 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgdkg\" (UniqueName: \"kubernetes.io/projected/75f58274-39a8-4c78-85bc-cbd464e896b5-kube-api-access-tgdkg\") pod \"sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p\" (UID: \"75f58274-39a8-4c78-85bc-cbd464e896b5\") " pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:31:08.665217 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.665169 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75f58274-39a8-4c78-85bc-cbd464e896b5-tmp-dir\") pod \"sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p\" (UID: \"75f58274-39a8-4c78-85bc-cbd464e896b5\") " pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:31:08.665442 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.665420 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75f58274-39a8-4c78-85bc-cbd464e896b5-home\") pod \"sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p\" (UID: \"75f58274-39a8-4c78-85bc-cbd464e896b5\") " pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:31:08.665509 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.665465 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75f58274-39a8-4c78-85bc-cbd464e896b5-model-cache\") pod \"sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p\" (UID: \"75f58274-39a8-4c78-85bc-cbd464e896b5\") " pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:31:08.665509 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.665487 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75f58274-39a8-4c78-85bc-cbd464e896b5-tmp-dir\") pod \"sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p\" (UID: \"75f58274-39a8-4c78-85bc-cbd464e896b5\") " pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:31:08.667081 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.667059 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75f58274-39a8-4c78-85bc-cbd464e896b5-dshm\") pod \"sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p\" (UID: \"75f58274-39a8-4c78-85bc-cbd464e896b5\") " pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:31:08.667354 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.667338 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75f58274-39a8-4c78-85bc-cbd464e896b5-tls-certs\") pod \"sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p\" (UID: \"75f58274-39a8-4c78-85bc-cbd464e896b5\") " pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:31:08.674421 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.674404 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgdkg\" (UniqueName: \"kubernetes.io/projected/75f58274-39a8-4c78-85bc-cbd464e896b5-kube-api-access-tgdkg\") pod \"sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p\" (UID: \"75f58274-39a8-4c78-85bc-cbd464e896b5\") " pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:31:08.872859 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:08.872826 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:31:09.146359 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:09.146335 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p"] Sep 14 11:31:09.148079 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:31:09.148050 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75f58274_39a8_4c78_85bc_cbd464e896b5.slice/crio-f540db3cbddb9a8a89c4967c8c31a727c40d6ad7ed048dc29052ff1ea2c7feae WatchSource:0}: Error finding container f540db3cbddb9a8a89c4967c8c31a727c40d6ad7ed048dc29052ff1ea2c7feae: Status 404 returned error can't find the container with id f540db3cbddb9a8a89c4967c8c31a727c40d6ad7ed048dc29052ff1ea2c7feae Sep 14 11:31:09.162715 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:09.162690 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" event={"ID":"75f58274-39a8-4c78-85bc-cbd464e896b5","Type":"ContainerStarted","Data":"f540db3cbddb9a8a89c4967c8c31a727c40d6ad7ed048dc29052ff1ea2c7feae"} Sep 14 11:31:10.169448 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:10.169412 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" event={"ID":"75f58274-39a8-4c78-85bc-cbd464e896b5","Type":"ContainerStarted","Data":"661b1ea4bbdcf942797a1661ed270bc3b689878418e435cf0731da0f81235d4c"} Sep 14 11:31:10.187384 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:10.187329 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" podStartSLOduration=2.187312357 podStartE2EDuration="2.187312357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:31:10.186027237 +0000 UTC m=+1828.249430144" watchObservedRunningTime="2026-09-14 11:31:10.187312357 +0000 UTC m=+1828.250715268" Sep 14 11:31:18.873926 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:18.873889 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:31:18.874316 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:18.873937 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:31:18.887550 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:18.887522 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:31:19.228264 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:19.228180 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:31:42.745755 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:31:42.745723 2556 scope.go:117] "RemoveContainer" containerID="b8c7a8aa12e9c63c35e5a523f7d88b993edb0d67718ce6db2fbc7ada6f426d75" Sep 14 11:32:25.478851 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:25.478810 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p"] Sep 14 11:32:40.498928 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:40.498861 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" podUID="75f58274-39a8-4c78-85bc-cbd464e896b5" containerName="main" containerID="cri-o://661b1ea4bbdcf942797a1661ed270bc3b689878418e435cf0731da0f81235d4c" gracePeriod=45 Sep 14 11:32:40.620823 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:40.620793 2556 generic.go:356] "Generic (PLEG): container finished" podID="75f58274-39a8-4c78-85bc-cbd464e896b5" containerID="661b1ea4bbdcf942797a1661ed270bc3b689878418e435cf0731da0f81235d4c" exitCode=0 Sep 14 11:32:40.620985 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:40.620877 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" event={"ID":"75f58274-39a8-4c78-85bc-cbd464e896b5","Type":"ContainerDied","Data":"661b1ea4bbdcf942797a1661ed270bc3b689878418e435cf0731da0f81235d4c"} Sep 14 11:32:40.754476 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:40.754405 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:32:40.849803 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:40.849766 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75f58274-39a8-4c78-85bc-cbd464e896b5-tmp-dir\") pod \"75f58274-39a8-4c78-85bc-cbd464e896b5\" (UID: \"75f58274-39a8-4c78-85bc-cbd464e896b5\") " Sep 14 11:32:40.849803 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:40.849815 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75f58274-39a8-4c78-85bc-cbd464e896b5-tls-certs\") pod \"75f58274-39a8-4c78-85bc-cbd464e896b5\" (UID: \"75f58274-39a8-4c78-85bc-cbd464e896b5\") " Sep 14 11:32:40.850076 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:40.849846 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75f58274-39a8-4c78-85bc-cbd464e896b5-model-cache\") pod \"75f58274-39a8-4c78-85bc-cbd464e896b5\" (UID: \"75f58274-39a8-4c78-85bc-cbd464e896b5\") " Sep 14 11:32:40.850076 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:40.849969 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75f58274-39a8-4c78-85bc-cbd464e896b5-home\") pod \"75f58274-39a8-4c78-85bc-cbd464e896b5\" (UID: \"75f58274-39a8-4c78-85bc-cbd464e896b5\") " Sep 14 11:32:40.850230 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:40.850111 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgdkg\" (UniqueName: \"kubernetes.io/projected/75f58274-39a8-4c78-85bc-cbd464e896b5-kube-api-access-tgdkg\") pod \"75f58274-39a8-4c78-85bc-cbd464e896b5\" (UID: \"75f58274-39a8-4c78-85bc-cbd464e896b5\") " Sep 14 11:32:40.850230 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:40.850147 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75f58274-39a8-4c78-85bc-cbd464e896b5-dshm\") pod \"75f58274-39a8-4c78-85bc-cbd464e896b5\" (UID: \"75f58274-39a8-4c78-85bc-cbd464e896b5\") " Sep 14 11:32:40.850230 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:40.850146 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f58274-39a8-4c78-85bc-cbd464e896b5-model-cache" (OuterVolumeSpecName: "model-cache") pod "75f58274-39a8-4c78-85bc-cbd464e896b5" (UID: "75f58274-39a8-4c78-85bc-cbd464e896b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:32:40.850230 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:40.850158 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f58274-39a8-4c78-85bc-cbd464e896b5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "75f58274-39a8-4c78-85bc-cbd464e896b5" (UID: "75f58274-39a8-4c78-85bc-cbd464e896b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:32:40.850230 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:40.850215 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f58274-39a8-4c78-85bc-cbd464e896b5-home" (OuterVolumeSpecName: "home") pod "75f58274-39a8-4c78-85bc-cbd464e896b5" (UID: "75f58274-39a8-4c78-85bc-cbd464e896b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:32:40.850419 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:40.850364 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75f58274-39a8-4c78-85bc-cbd464e896b5-model-cache\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:32:40.850419 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:40.850376 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75f58274-39a8-4c78-85bc-cbd464e896b5-home\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:32:40.850419 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:40.850388 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75f58274-39a8-4c78-85bc-cbd464e896b5-tmp-dir\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:32:40.852257 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:40.852229 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f58274-39a8-4c78-85bc-cbd464e896b5-dshm" (OuterVolumeSpecName: "dshm") pod "75f58274-39a8-4c78-85bc-cbd464e896b5" (UID: "75f58274-39a8-4c78-85bc-cbd464e896b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:32:40.852257 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:40.852242 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f58274-39a8-4c78-85bc-cbd464e896b5-kube-api-access-tgdkg" (OuterVolumeSpecName: "kube-api-access-tgdkg") pod "75f58274-39a8-4c78-85bc-cbd464e896b5" (UID: "75f58274-39a8-4c78-85bc-cbd464e896b5"). InnerVolumeSpecName "kube-api-access-tgdkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:32:40.852403 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:40.852319 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f58274-39a8-4c78-85bc-cbd464e896b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75f58274-39a8-4c78-85bc-cbd464e896b5" (UID: "75f58274-39a8-4c78-85bc-cbd464e896b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:32:40.951397 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:40.951354 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgdkg\" (UniqueName: \"kubernetes.io/projected/75f58274-39a8-4c78-85bc-cbd464e896b5-kube-api-access-tgdkg\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:32:40.951397 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:40.951394 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75f58274-39a8-4c78-85bc-cbd464e896b5-dshm\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:32:40.951397 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:40.951405 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75f58274-39a8-4c78-85bc-cbd464e896b5-tls-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:32:41.626904 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:41.626876 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" Sep 14 11:32:41.627440 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:41.626922 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p" event={"ID":"75f58274-39a8-4c78-85bc-cbd464e896b5","Type":"ContainerDied","Data":"f540db3cbddb9a8a89c4967c8c31a727c40d6ad7ed048dc29052ff1ea2c7feae"} Sep 14 11:32:41.627440 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:41.626972 2556 scope.go:117] "RemoveContainer" containerID="661b1ea4bbdcf942797a1661ed270bc3b689878418e435cf0731da0f81235d4c" Sep 14 11:32:41.647560 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:41.647529 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p"] Sep 14 11:32:41.651381 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:41.651352 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-55c025/sub-test-model-55c025-kserve-59cd6fb9dc-9dd7p"] Sep 14 11:32:42.382351 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:32:42.382311 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75f58274-39a8-4c78-85bc-cbd464e896b5" path="/var/lib/kubelet/pods/75f58274-39a8-4c78-85bc-cbd464e896b5/volumes" Sep 14 11:33:21.347727 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.347691 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr"] Sep 14 11:33:21.348178 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.348020 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f58274-39a8-4c78-85bc-cbd464e896b5" containerName="main" Sep 14 11:33:21.348178 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.348034 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f58274-39a8-4c78-85bc-cbd464e896b5" containerName="main" Sep 14 11:33:21.348178 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.348114 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="75f58274-39a8-4c78-85bc-cbd464e896b5" containerName="main" Sep 14 11:33:21.350751 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.350730 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.353436 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.353412 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-df3763-openshift-default-dockercfg-bjrfr\"" Sep 14 11:33:21.366625 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.366600 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr"] Sep 14 11:33:21.493188 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.493148 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istio-podinfo\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.493374 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.493209 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istiod-ca-cert\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.493374 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.493296 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-workload-socket\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.493374 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.493341 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istio-token\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.493374 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.493372 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istio-envoy\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.493538 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.493402 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-workload-certs\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.493538 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.493440 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istio-data\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.493538 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.493462 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-credential-socket\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.493538 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.493506 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt68z\" (UniqueName: \"kubernetes.io/projected/4052cd69-fa63-4d4c-a598-91fa8588ef5d-kube-api-access-nt68z\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.594209 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.594177 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istiod-ca-cert\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.594400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.594229 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-workload-socket\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.594400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.594257 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istio-token\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.594400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.594289 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istio-envoy\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.594400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.594314 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-workload-certs\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.594400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.594358 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istio-data\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.594400 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.594382 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-credential-socket\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.594715 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.594418 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt68z\" (UniqueName: \"kubernetes.io/projected/4052cd69-fa63-4d4c-a598-91fa8588ef5d-kube-api-access-nt68z\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.594715 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.594479 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istio-podinfo\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.594823 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.594722 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-workload-certs\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.594915 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.594891 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-workload-socket\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.594988 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.594931 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istio-data\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.595055 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.595018 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istiod-ca-cert\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.595196 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.595175 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-credential-socket\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.596743 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.596709 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istio-envoy\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.597050 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.597032 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istio-podinfo\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.608622 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.608600 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istio-token\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.614052 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.614025 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt68z\" (UniqueName: \"kubernetes.io/projected/4052cd69-fa63-4d4c-a598-91fa8588ef5d-kube-api-access-nt68z\") pod \"e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.661048 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.661018 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:21.936533 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.936505 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr"] Sep 14 11:33:21.938559 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:33:21.938533 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4052cd69_fa63_4d4c_a598_91fa8588ef5d.slice/crio-a0b9a8dce16a5a78d7d1d229fcec42ba31e9f5b3a480aa1441c0a9ed06d90735 WatchSource:0}: Error finding container a0b9a8dce16a5a78d7d1d229fcec42ba31e9f5b3a480aa1441c0a9ed06d90735: Status 404 returned error can't find the container with id a0b9a8dce16a5a78d7d1d229fcec42ba31e9f5b3a480aa1441c0a9ed06d90735 Sep 14 11:33:21.940892 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.940862 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:33:21.940951 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.940924 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:33:21.943121 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:21.941330 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:33:22.831864 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:22.831829 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" event={"ID":"4052cd69-fa63-4d4c-a598-91fa8588ef5d","Type":"ContainerStarted","Data":"0dc12bd05847cd9b5108046c7a0659dd86feebefe05ea73dd758372e93f0c6f7"} Sep 14 11:33:22.832265 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:22.831885 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" event={"ID":"4052cd69-fa63-4d4c-a598-91fa8588ef5d","Type":"ContainerStarted","Data":"a0b9a8dce16a5a78d7d1d229fcec42ba31e9f5b3a480aa1441c0a9ed06d90735"} Sep 14 11:33:22.854312 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:22.854265 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" podStartSLOduration=1.854251316 podStartE2EDuration="1.854251316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:33:22.851565619 +0000 UTC m=+1960.914968596" watchObservedRunningTime="2026-09-14 11:33:22.854251316 +0000 UTC m=+1960.917654228" Sep 14 11:33:23.661731 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:23.661689 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:23.666640 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:23.666615 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:23.836462 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:23.836435 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:23.837296 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:23.837279 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:26.749615 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:26.749581 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr"] Sep 14 11:33:26.752972 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:26.752955 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr" Sep 14 11:33:26.755196 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:26.755175 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-df3763-dockercfg-sf2rh\"" Sep 14 11:33:26.755307 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:26.755214 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-df3763\"" Sep 14 11:33:26.763780 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:26.763754 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr"] Sep 14 11:33:26.784339 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:26.784304 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf"] Sep 14 11:33:26.788185 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:26.788161 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf" Sep 14 11:33:26.797231 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:26.797207 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf"] Sep 14 11:33:26.843250 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:26.843210 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/365bafbd-8258-4168-a55d-0d6a4667e47a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr\" (UID: \"365bafbd-8258-4168-a55d-0d6a4667e47a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr" Sep 14 11:33:26.843675 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:26.843277 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjzn6\" (UniqueName: \"kubernetes.io/projected/365bafbd-8258-4168-a55d-0d6a4667e47a-kube-api-access-cjzn6\") pod \"payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr\" (UID: \"365bafbd-8258-4168-a55d-0d6a4667e47a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr" Sep 14 11:33:26.945134 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:26.945066 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjzn6\" (UniqueName: \"kubernetes.io/projected/365bafbd-8258-4168-a55d-0d6a4667e47a-kube-api-access-cjzn6\") pod \"payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr\" (UID: \"365bafbd-8258-4168-a55d-0d6a4667e47a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr" Sep 14 11:33:26.945339 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:26.945169 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf\" (UID: \"3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf" Sep 14 11:33:26.945339 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:26.945192 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjgfv\" (UniqueName: \"kubernetes.io/projected/3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58-kube-api-access-zjgfv\") pod \"payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf\" (UID: \"3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf" Sep 14 11:33:26.945339 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:26.945239 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/365bafbd-8258-4168-a55d-0d6a4667e47a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr\" (UID: \"365bafbd-8258-4168-a55d-0d6a4667e47a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr" Sep 14 11:33:26.945880 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:26.945858 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/365bafbd-8258-4168-a55d-0d6a4667e47a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr\" (UID: \"365bafbd-8258-4168-a55d-0d6a4667e47a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr" Sep 14 11:33:26.953405 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:26.953381 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjzn6\" (UniqueName: \"kubernetes.io/projected/365bafbd-8258-4168-a55d-0d6a4667e47a-kube-api-access-cjzn6\") pod \"payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr\" (UID: \"365bafbd-8258-4168-a55d-0d6a4667e47a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr" Sep 14 11:33:27.046529 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:27.046435 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf\" (UID: \"3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf" Sep 14 11:33:27.046529 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:27.046475 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjgfv\" (UniqueName: \"kubernetes.io/projected/3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58-kube-api-access-zjgfv\") pod \"payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf\" (UID: \"3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf" Sep 14 11:33:27.047183 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:27.047164 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf\" (UID: \"3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf" Sep 14 11:33:27.053980 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:27.053957 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjgfv\" (UniqueName: \"kubernetes.io/projected/3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58-kube-api-access-zjgfv\") pod \"payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf\" (UID: \"3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf" Sep 14 11:33:27.064815 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:27.064793 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr" Sep 14 11:33:27.100153 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:27.100119 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf" Sep 14 11:33:27.340168 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:27.340140 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr"] Sep 14 11:33:27.341806 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:33:27.341775 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod365bafbd_8258_4168_a55d_0d6a4667e47a.slice/crio-d84934c27dd23b3be236d435c01fb5ad8e076772454275c0bcab35b93c2e06ac WatchSource:0}: Error finding container d84934c27dd23b3be236d435c01fb5ad8e076772454275c0bcab35b93c2e06ac: Status 404 returned error can't find the container with id d84934c27dd23b3be236d435c01fb5ad8e076772454275c0bcab35b93c2e06ac Sep 14 11:33:27.382740 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:27.382708 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf"] Sep 14 11:33:27.384066 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:33:27.384025 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba63976_95cb_4b46_8bb0_4dbc3f2bfa58.slice/crio-6d0fc536e336e07af246fc0b2b5bca3c5b5c8af67c18b0681db8835fe51c201e WatchSource:0}: Error finding container 6d0fc536e336e07af246fc0b2b5bca3c5b5c8af67c18b0681db8835fe51c201e: Status 404 returned error can't find the container with id 6d0fc536e336e07af246fc0b2b5bca3c5b5c8af67c18b0681db8835fe51c201e Sep 14 11:33:27.858657 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:27.858618 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf" event={"ID":"3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58","Type":"ContainerStarted","Data":"9ea43572efb56db0c49e9fff922e22fec1a096dcd7fe8f5fcee89597a2825ecd"} Sep 14 11:33:27.859139 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:27.858683 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf" event={"ID":"3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58","Type":"ContainerStarted","Data":"6d0fc536e336e07af246fc0b2b5bca3c5b5c8af67c18b0681db8835fe51c201e"} Sep 14 11:33:27.859139 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:27.858717 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf" Sep 14 11:33:27.860208 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:27.860179 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr" event={"ID":"365bafbd-8258-4168-a55d-0d6a4667e47a","Type":"ContainerStarted","Data":"e2c2616d12bbc7ddbfdffd454d0af44190f36ffca03fd28289ad33b0dc8ad6ad"} Sep 14 11:33:27.860324 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:27.860234 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr" Sep 14 11:33:27.860324 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:27.860257 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr" event={"ID":"365bafbd-8258-4168-a55d-0d6a4667e47a","Type":"ContainerStarted","Data":"d84934c27dd23b3be236d435c01fb5ad8e076772454275c0bcab35b93c2e06ac"} Sep 14 11:33:27.875012 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:27.874948 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf" podStartSLOduration=1.8749295689999999 podStartE2EDuration="1.874929569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:33:27.874570979 +0000 UTC m=+1965.937973891" watchObservedRunningTime="2026-09-14 11:33:27.874929569 +0000 UTC m=+1965.938332481" Sep 14 11:33:27.893880 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:27.893827 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr" podStartSLOduration=1.893813009 podStartE2EDuration="1.893813009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:33:27.892060665 +0000 UTC m=+1965.955463575" watchObservedRunningTime="2026-09-14 11:33:27.893813009 +0000 UTC m=+1965.957215919" Sep 14 11:33:38.867498 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:38.867464 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf" Sep 14 11:33:38.867945 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:38.867539 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr" Sep 14 11:33:39.878245 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:39.878202 2556 patch_prober.go:28] interesting pod/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 14 11:33:39.878684 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:39.878267 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" podUID="4052cd69-fa63-4d4c-a598-91fa8588ef5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 14 11:33:41.967983 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:41.967949 2556 generic.go:356] "Generic (PLEG): container finished" podID="4052cd69-fa63-4d4c-a598-91fa8588ef5d" containerID="0dc12bd05847cd9b5108046c7a0659dd86feebefe05ea73dd758372e93f0c6f7" exitCode=0 Sep 14 11:33:41.968412 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:41.968006 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" event={"ID":"4052cd69-fa63-4d4c-a598-91fa8588ef5d","Type":"ContainerDied","Data":"0dc12bd05847cd9b5108046c7a0659dd86feebefe05ea73dd758372e93f0c6f7"} Sep 14 11:33:41.968412 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:41.968377 2556 scope.go:117] "RemoveContainer" containerID="0dc12bd05847cd9b5108046c7a0659dd86feebefe05ea73dd758372e93f0c6f7" Sep 14 11:33:41.969397 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:41.969366 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:33:41.969447 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:41.969434 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:33:41.969485 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:41.969464 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:33:42.974987 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:42.974942 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" event={"ID":"4052cd69-fa63-4d4c-a598-91fa8588ef5d","Type":"ContainerStarted","Data":"19e40a3c5454bf1f2a65708efe5160141e14022888874194fd1e06e2c02d8b88"} Sep 14 11:33:43.661746 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:43.661712 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:43.661922 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:43.661878 2556 patch_prober.go:28] interesting pod/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 14 11:33:43.661969 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:43.661925 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" podUID="4052cd69-fa63-4d4c-a598-91fa8588ef5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 14 11:33:44.662123 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:44.662067 2556 patch_prober.go:28] interesting pod/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 14 11:33:44.662494 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:44.662155 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" podUID="4052cd69-fa63-4d4c-a598-91fa8588ef5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 14 11:33:45.661931 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:45.661896 2556 patch_prober.go:28] interesting pod/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 14 11:33:45.662129 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:45.661950 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" podUID="4052cd69-fa63-4d4c-a598-91fa8588ef5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 14 11:33:46.661754 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:46.661713 2556 patch_prober.go:28] interesting pod/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 14 11:33:46.662157 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:46.661770 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" podUID="4052cd69-fa63-4d4c-a598-91fa8588ef5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 14 11:33:47.661532 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:47.661494 2556 patch_prober.go:28] interesting pod/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 14 11:33:47.661714 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:47.661590 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" podUID="4052cd69-fa63-4d4c-a598-91fa8588ef5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 14 11:33:48.664550 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:48.664521 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:48.665025 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:48.664925 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:48.665608 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:48.665586 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:33:56.571060 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:56.571023 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42"] Sep 14 11:33:56.574494 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:56.574477 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42" Sep 14 11:33:56.577100 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:56.577073 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d0f937-dockercfg-2dktj\"" Sep 14 11:33:56.577231 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:56.577133 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d0f937\"" Sep 14 11:33:56.585540 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:56.585517 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42"] Sep 14 11:33:56.609221 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:56.609197 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jknnk\" (UniqueName: \"kubernetes.io/projected/69443040-eb9a-4a5e-8783-40b17474557b-kube-api-access-jknnk\") pod \"payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42\" (UID: \"69443040-eb9a-4a5e-8783-40b17474557b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42" Sep 14 11:33:56.609356 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:56.609260 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69443040-eb9a-4a5e-8783-40b17474557b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42\" (UID: \"69443040-eb9a-4a5e-8783-40b17474557b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42" Sep 14 11:33:56.710322 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:56.710281 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jknnk\" (UniqueName: \"kubernetes.io/projected/69443040-eb9a-4a5e-8783-40b17474557b-kube-api-access-jknnk\") pod \"payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42\" (UID: \"69443040-eb9a-4a5e-8783-40b17474557b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42" Sep 14 11:33:56.710504 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:56.710381 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69443040-eb9a-4a5e-8783-40b17474557b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42\" (UID: \"69443040-eb9a-4a5e-8783-40b17474557b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42" Sep 14 11:33:56.711081 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:56.711060 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69443040-eb9a-4a5e-8783-40b17474557b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42\" (UID: \"69443040-eb9a-4a5e-8783-40b17474557b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42" Sep 14 11:33:56.718239 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:56.718213 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jknnk\" (UniqueName: \"kubernetes.io/projected/69443040-eb9a-4a5e-8783-40b17474557b-kube-api-access-jknnk\") pod \"payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42\" (UID: \"69443040-eb9a-4a5e-8783-40b17474557b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42" Sep 14 11:33:56.885672 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:56.885638 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42" Sep 14 11:33:57.155220 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:57.155193 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42"] Sep 14 11:33:57.157198 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:33:57.157169 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69443040_eb9a_4a5e_8783_40b17474557b.slice/crio-f8b287c465967dac12eb709a703f855442febe0da1d26b82a5bd8216a60e645c WatchSource:0}: Error finding container f8b287c465967dac12eb709a703f855442febe0da1d26b82a5bd8216a60e645c: Status 404 returned error can't find the container with id f8b287c465967dac12eb709a703f855442febe0da1d26b82a5bd8216a60e645c Sep 14 11:33:58.060050 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:58.060013 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42" event={"ID":"69443040-eb9a-4a5e-8783-40b17474557b","Type":"ContainerStarted","Data":"1bca724dc9b743894eec37dfcfef976de354bbf89eb8a1f10b82f270213bbec9"} Sep 14 11:33:58.060439 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:58.060063 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42" event={"ID":"69443040-eb9a-4a5e-8783-40b17474557b","Type":"ContainerStarted","Data":"f8b287c465967dac12eb709a703f855442febe0da1d26b82a5bd8216a60e645c"} Sep 14 11:33:58.060439 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:58.060137 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42" Sep 14 11:33:58.080279 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:33:58.080236 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42" podStartSLOduration=2.080223051 podStartE2EDuration="2.080223051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:33:58.078452573 +0000 UTC m=+1996.141855497" watchObservedRunningTime="2026-09-14 11:33:58.080223051 +0000 UTC m=+1996.143625962" Sep 14 11:34:09.066539 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:09.066510 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42" Sep 14 11:34:23.600984 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:23.600957 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42_69443040-eb9a-4a5e-8783-40b17474557b/payload-processing/0.log" Sep 14 11:34:49.201629 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:49.201596 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf_3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58/payload-processing/0.log" Sep 14 11:34:49.468158 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:49.468062 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42_69443040-eb9a-4a5e-8783-40b17474557b/payload-processing/0.log" Sep 14 11:34:50.053261 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.053230 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9"] Sep 14 11:34:50.056429 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.056412 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.058866 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.058847 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-1ec353-openshift-default-dockercfg-z4w7p\"" Sep 14 11:34:50.072039 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.072016 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9"] Sep 14 11:34:50.162512 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.162481 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-istio-data\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.162512 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.162517 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-credential-socket\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.162742 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.162544 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzswv\" (UniqueName: \"kubernetes.io/projected/e8bc5265-7b66-4876-b29d-1376c7d7d576-kube-api-access-hzswv\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.162742 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.162607 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bc5265-7b66-4876-b29d-1376c7d7d576-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.162742 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.162629 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-workload-socket\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.162742 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.162661 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-istio-envoy\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.162742 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.162715 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-workload-certs\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.162742 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.162736 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bc5265-7b66-4876-b29d-1376c7d7d576-istio-podinfo\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.162947 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.162773 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bc5265-7b66-4876-b29d-1376c7d7d576-istio-token\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.263791 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.263760 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-credential-socket\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.264257 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.263794 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzswv\" (UniqueName: \"kubernetes.io/projected/e8bc5265-7b66-4876-b29d-1376c7d7d576-kube-api-access-hzswv\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.264257 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.263823 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bc5265-7b66-4876-b29d-1376c7d7d576-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.264257 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.263842 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-workload-socket\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.264257 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.263872 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-istio-envoy\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.264257 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.263895 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-workload-certs\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.264257 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.263914 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bc5265-7b66-4876-b29d-1376c7d7d576-istio-podinfo\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.264257 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.263942 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bc5265-7b66-4876-b29d-1376c7d7d576-istio-token\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.264257 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.263995 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-istio-data\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.264597 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.264270 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-credential-socket\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.264597 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.264334 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-istio-data\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.264597 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.264356 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-workload-socket\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.264597 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.264561 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-workload-certs\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.264597 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.264577 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bc5265-7b66-4876-b29d-1376c7d7d576-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.266585 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.266555 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-istio-envoy\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.266705 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.266686 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bc5265-7b66-4876-b29d-1376c7d7d576-istio-podinfo\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.272577 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.272553 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bc5265-7b66-4876-b29d-1376c7d7d576-istio-token\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.272666 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.272613 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzswv\" (UniqueName: \"kubernetes.io/projected/e8bc5265-7b66-4876-b29d-1376c7d7d576-kube-api-access-hzswv\") pod \"e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.366668 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.366645 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:50.641860 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.641812 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9"] Sep 14 11:34:50.644022 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:34:50.643993 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8bc5265_7b66_4876_b29d_1376c7d7d576.slice/crio-d54fa044b7b46fd8d171857a4a1d312b95c8f712b644352347ad49d295b0637c WatchSource:0}: Error finding container d54fa044b7b46fd8d171857a4a1d312b95c8f712b644352347ad49d295b0637c: Status 404 returned error can't find the container with id d54fa044b7b46fd8d171857a4a1d312b95c8f712b644352347ad49d295b0637c Sep 14 11:34:50.646185 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.646149 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:34:50.646261 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.646214 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:34:50.646261 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:50.646242 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:34:51.336628 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:51.336596 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" event={"ID":"e8bc5265-7b66-4876-b29d-1376c7d7d576","Type":"ContainerStarted","Data":"05afe2f9644373ba9cc6cd6e50c9f593a81782c26812f7a2d9b5f4b4a519c2e7"} Sep 14 11:34:51.336979 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:51.336652 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" event={"ID":"e8bc5265-7b66-4876-b29d-1376c7d7d576","Type":"ContainerStarted","Data":"d54fa044b7b46fd8d171857a4a1d312b95c8f712b644352347ad49d295b0637c"} Sep 14 11:34:51.357885 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:51.357841 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" podStartSLOduration=1.357827247 podStartE2EDuration="1.357827247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:34:51.356346463 +0000 UTC m=+2049.419749372" watchObservedRunningTime="2026-09-14 11:34:51.357827247 +0000 UTC m=+2049.421230156" Sep 14 11:34:51.367388 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:51.367365 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:51.372514 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:51.372495 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:52.341936 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:52.341907 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:34:52.342777 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:34:52.342759 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:35:11.448716 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:11.448683 2556 generic.go:356] "Generic (PLEG): container finished" podID="e8bc5265-7b66-4876-b29d-1376c7d7d576" containerID="05afe2f9644373ba9cc6cd6e50c9f593a81782c26812f7a2d9b5f4b4a519c2e7" exitCode=0 Sep 14 11:35:11.449068 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:11.448751 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" event={"ID":"e8bc5265-7b66-4876-b29d-1376c7d7d576","Type":"ContainerDied","Data":"05afe2f9644373ba9cc6cd6e50c9f593a81782c26812f7a2d9b5f4b4a519c2e7"} Sep 14 11:35:11.449138 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:11.449115 2556 scope.go:117] "RemoveContainer" containerID="05afe2f9644373ba9cc6cd6e50c9f593a81782c26812f7a2d9b5f4b4a519c2e7" Sep 14 11:35:11.450188 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:11.450161 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:35:11.450272 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:11.450257 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:35:11.450313 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:11.450292 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 14 11:35:12.455946 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:12.455912 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" event={"ID":"e8bc5265-7b66-4876-b29d-1376c7d7d576","Type":"ContainerStarted","Data":"42586f5fa80d0e6f0d231de63e58458cafb8b0a899eaf0ca89f5ae05cef8018f"} Sep 14 11:35:13.367379 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:13.367350 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:35:13.367581 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:13.367558 2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Sep 14 11:35:13.367636 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:13.367616 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" podUID="e8bc5265-7b66-4876-b29d-1376c7d7d576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Sep 14 11:35:14.367084 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:14.367051 2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Sep 14 11:35:14.367501 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:14.367129 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" podUID="e8bc5265-7b66-4876-b29d-1376c7d7d576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Sep 14 11:35:15.367831 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:15.367793 2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Sep 14 11:35:15.368231 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:15.367855 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" podUID="e8bc5265-7b66-4876-b29d-1376c7d7d576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Sep 14 11:35:16.368022 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:16.367984 2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Sep 14 11:35:16.368421 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:16.368041 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" podUID="e8bc5265-7b66-4876-b29d-1376c7d7d576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Sep 14 11:35:17.370702 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:17.370665 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:35:17.371151 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:17.370915 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:35:17.371533 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:17.371518 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:35:24.007613 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:24.007583 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ec353-cqqcz"] Sep 14 11:35:24.010144 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:24.010128 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ec353-cqqcz" Sep 14 11:35:24.012269 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:24.012229 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 11:35:24.012394 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:24.012292 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f98vt\"" Sep 14 11:35:24.019191 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:24.019171 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ec353-cqqcz"] Sep 14 11:35:24.073737 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:24.073705 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cd69443-b778-4151-a606-6ff514e6bbc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-1ec353-cqqcz\" (UID: \"2cd69443-b778-4151-a606-6ff514e6bbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ec353-cqqcz" Sep 14 11:35:24.174424 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:24.174397 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cd69443-b778-4151-a606-6ff514e6bbc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-1ec353-cqqcz\" (UID: \"2cd69443-b778-4151-a606-6ff514e6bbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ec353-cqqcz" Sep 14 11:35:24.174954 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:24.174934 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cd69443-b778-4151-a606-6ff514e6bbc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-1ec353-cqqcz\" (UID: \"2cd69443-b778-4151-a606-6ff514e6bbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ec353-cqqcz" Sep 14 11:35:24.321045 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:24.320978 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ec353-cqqcz" Sep 14 11:35:24.588554 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:24.588533 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ec353-cqqcz"] Sep 14 11:35:24.590653 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:35:24.590626 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cd69443_b778_4151_a606_6ff514e6bbc6.slice/crio-7603880eb0a80992b2034b5986b649ca617ac3f4a94e099ceaec3dd6cf485cf0 WatchSource:0}: Error finding container 7603880eb0a80992b2034b5986b649ca617ac3f4a94e099ceaec3dd6cf485cf0: Status 404 returned error can't find the container with id 7603880eb0a80992b2034b5986b649ca617ac3f4a94e099ceaec3dd6cf485cf0 Sep 14 11:35:25.521964 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:25.521927 2556 generic.go:356] "Generic (PLEG): container finished" podID="2cd69443-b778-4151-a606-6ff514e6bbc6" containerID="a43502a11b469cef48275ca7b54a0351c30e5cafd66c6fe568bb28c44ec37602" exitCode=0 Sep 14 11:35:25.522417 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:25.521980 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ec353-cqqcz" event={"ID":"2cd69443-b778-4151-a606-6ff514e6bbc6","Type":"ContainerDied","Data":"a43502a11b469cef48275ca7b54a0351c30e5cafd66c6fe568bb28c44ec37602"} Sep 14 11:35:25.522417 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:25.522032 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ec353-cqqcz" event={"ID":"2cd69443-b778-4151-a606-6ff514e6bbc6","Type":"ContainerStarted","Data":"7603880eb0a80992b2034b5986b649ca617ac3f4a94e099ceaec3dd6cf485cf0"} Sep 14 11:35:26.654445 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:26.654422 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ec353-cqqcz" Sep 14 11:35:26.696479 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:26.696451 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cd69443-b778-4151-a606-6ff514e6bbc6-maas-api-service-ca\") pod \"2cd69443-b778-4151-a606-6ff514e6bbc6\" (UID: \"2cd69443-b778-4151-a606-6ff514e6bbc6\") " Sep 14 11:35:26.696777 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:26.696754 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cd69443-b778-4151-a606-6ff514e6bbc6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2cd69443-b778-4151-a606-6ff514e6bbc6" (UID: "2cd69443-b778-4151-a606-6ff514e6bbc6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:35:26.797145 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:26.797066 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cd69443-b778-4151-a606-6ff514e6bbc6-maas-api-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:35:27.533034 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:27.533004 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ec353-cqqcz" Sep 14 11:35:27.533230 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:27.533039 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ec353-cqqcz" event={"ID":"2cd69443-b778-4151-a606-6ff514e6bbc6","Type":"ContainerDied","Data":"7603880eb0a80992b2034b5986b649ca617ac3f4a94e099ceaec3dd6cf485cf0"} Sep 14 11:35:27.533230 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:27.533084 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7603880eb0a80992b2034b5986b649ca617ac3f4a94e099ceaec3dd6cf485cf0" Sep 14 11:35:35.122149 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:35.122111 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ec353-cqqcz"] Sep 14 11:35:35.123835 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:35.123813 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ec353-cqqcz"] Sep 14 11:35:36.382834 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:36.382796 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cd69443-b778-4151-a606-6ff514e6bbc6" path="/var/lib/kubelet/pods/2cd69443-b778-4151-a606-6ff514e6bbc6/volumes" Sep 14 11:35:42.029857 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:42.029821 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9"] Sep 14 11:35:42.030340 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:42.030271 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" podUID="e8bc5265-7b66-4876-b29d-1376c7d7d576" containerName="istio-proxy" containerID="cri-o://42586f5fa80d0e6f0d231de63e58458cafb8b0a899eaf0ca89f5ae05cef8018f" gracePeriod=30 Sep 14 11:35:42.792107 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:42.792062 2556 scope.go:117] "RemoveContainer" containerID="efc5b263c3806fb9bd79589b676cd9dad6891c2e6e9b03fc92b6fe4f6e1ee478" Sep 14 11:35:42.801197 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:42.801172 2556 scope.go:117] "RemoveContainer" containerID="085f46519a8179cdeca475cbaaa328ccd56b4d5686a9048408e448d3db58beef" Sep 14 11:35:42.808596 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:42.808578 2556 scope.go:117] "RemoveContainer" containerID="48d0f7503ee406280c3b308fa6ce649c7b596ef2981284d91fab94cae4e29bce" Sep 14 11:35:45.519117 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:45.519068 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-df3763-8c2gp"] Sep 14 11:35:45.519610 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:45.519592 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cd69443-b778-4151-a606-6ff514e6bbc6" containerName="revoke-keys" Sep 14 11:35:45.519659 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:45.519616 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cd69443-b778-4151-a606-6ff514e6bbc6" containerName="revoke-keys" Sep 14 11:35:45.519706 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:45.519696 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cd69443-b778-4151-a606-6ff514e6bbc6" containerName="revoke-keys" Sep 14 11:35:45.522925 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:45.522909 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-df3763-8c2gp" Sep 14 11:35:45.525334 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:45.525315 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 11:35:45.525432 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:45.525339 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f98vt\"" Sep 14 11:35:45.529358 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:45.529332 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-df3763-8c2gp"] Sep 14 11:35:45.664395 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:45.664362 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a339534-7e42-4f4d-b4d5-52e81d594cfc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-df3763-8c2gp\" (UID: \"0a339534-7e42-4f4d-b4d5-52e81d594cfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-df3763-8c2gp" Sep 14 11:35:45.765193 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:45.765163 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a339534-7e42-4f4d-b4d5-52e81d594cfc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-df3763-8c2gp\" (UID: \"0a339534-7e42-4f4d-b4d5-52e81d594cfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-df3763-8c2gp" Sep 14 11:35:45.765765 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:45.765745 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a339534-7e42-4f4d-b4d5-52e81d594cfc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-df3763-8c2gp\" (UID: \"0a339534-7e42-4f4d-b4d5-52e81d594cfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-df3763-8c2gp" Sep 14 11:35:45.834107 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:45.834024 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-df3763-8c2gp" Sep 14 11:35:46.105138 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:46.105108 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-df3763-8c2gp"] Sep 14 11:35:46.106600 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:35:46.106572 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a339534_7e42_4f4d_b4d5_52e81d594cfc.slice/crio-a7a3cbcfb2518ff245539baf1a90698f5270f1d6f98e286d3915382231cb90a6 WatchSource:0}: Error finding container a7a3cbcfb2518ff245539baf1a90698f5270f1d6f98e286d3915382231cb90a6: Status 404 returned error can't find the container with id a7a3cbcfb2518ff245539baf1a90698f5270f1d6f98e286d3915382231cb90a6 Sep 14 11:35:46.632389 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:46.632354 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-df3763-8c2gp" event={"ID":"0a339534-7e42-4f4d-b4d5-52e81d594cfc","Type":"ContainerStarted","Data":"a800e48dc3c460ead0f71d3207eccae028d760ee93747e2c4ef9a1c990c53be6"} Sep 14 11:35:46.632747 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:46.632410 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-df3763-8c2gp" event={"ID":"0a339534-7e42-4f4d-b4d5-52e81d594cfc","Type":"ContainerStarted","Data":"a7a3cbcfb2518ff245539baf1a90698f5270f1d6f98e286d3915382231cb90a6"} Sep 14 11:35:46.650817 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:46.650723 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-df3763-8c2gp" podStartSLOduration=1.650708688 podStartE2EDuration="1.650708688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:35:46.64852574 +0000 UTC m=+2104.711928652" watchObservedRunningTime="2026-09-14 11:35:46.650708688 +0000 UTC m=+2104.714111597" Sep 14 11:35:47.269416 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.269388 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:35:47.383063 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.383030 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-credential-socket\") pod \"e8bc5265-7b66-4876-b29d-1376c7d7d576\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " Sep 14 11:35:47.383261 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.383098 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bc5265-7b66-4876-b29d-1376c7d7d576-istiod-ca-cert\") pod \"e8bc5265-7b66-4876-b29d-1376c7d7d576\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " Sep 14 11:35:47.383261 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.383136 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-istio-data\") pod \"e8bc5265-7b66-4876-b29d-1376c7d7d576\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " Sep 14 11:35:47.383261 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.383157 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzswv\" (UniqueName: \"kubernetes.io/projected/e8bc5265-7b66-4876-b29d-1376c7d7d576-kube-api-access-hzswv\") pod \"e8bc5265-7b66-4876-b29d-1376c7d7d576\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " Sep 14 11:35:47.383261 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.383218 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-workload-socket\") pod \"e8bc5265-7b66-4876-b29d-1376c7d7d576\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " Sep 14 11:35:47.383261 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.383247 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-istio-envoy\") pod \"e8bc5265-7b66-4876-b29d-1376c7d7d576\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " Sep 14 11:35:47.383534 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.383313 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bc5265-7b66-4876-b29d-1376c7d7d576-istio-token\") pod \"e8bc5265-7b66-4876-b29d-1376c7d7d576\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " Sep 14 11:35:47.383534 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.383346 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bc5265-7b66-4876-b29d-1376c7d7d576-istio-podinfo\") pod \"e8bc5265-7b66-4876-b29d-1376c7d7d576\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " Sep 14 11:35:47.383534 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.383371 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-workload-certs\") pod \"e8bc5265-7b66-4876-b29d-1376c7d7d576\" (UID: \"e8bc5265-7b66-4876-b29d-1376c7d7d576\") " Sep 14 11:35:47.383751 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.383337 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e8bc5265-7b66-4876-b29d-1376c7d7d576" (UID: "e8bc5265-7b66-4876-b29d-1376c7d7d576"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:35:47.383812 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.383697 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:35:47.383812 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.383768 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e8bc5265-7b66-4876-b29d-1376c7d7d576" (UID: "e8bc5265-7b66-4876-b29d-1376c7d7d576"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:35:47.383812 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.383488 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-istio-data" (OuterVolumeSpecName: "istio-data") pod "e8bc5265-7b66-4876-b29d-1376c7d7d576" (UID: "e8bc5265-7b66-4876-b29d-1376c7d7d576"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:35:47.383812 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.383575 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8bc5265-7b66-4876-b29d-1376c7d7d576-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e8bc5265-7b66-4876-b29d-1376c7d7d576" (UID: "e8bc5265-7b66-4876-b29d-1376c7d7d576"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:35:47.384024 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.383971 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e8bc5265-7b66-4876-b29d-1376c7d7d576" (UID: "e8bc5265-7b66-4876-b29d-1376c7d7d576"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:35:47.385635 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.385607 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8bc5265-7b66-4876-b29d-1376c7d7d576-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e8bc5265-7b66-4876-b29d-1376c7d7d576" (UID: "e8bc5265-7b66-4876-b29d-1376c7d7d576"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 11:35:47.385791 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.385768 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8bc5265-7b66-4876-b29d-1376c7d7d576-istio-token" (OuterVolumeSpecName: "istio-token") pod "e8bc5265-7b66-4876-b29d-1376c7d7d576" (UID: "e8bc5265-7b66-4876-b29d-1376c7d7d576"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:35:47.385908 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.385890 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8bc5265-7b66-4876-b29d-1376c7d7d576-kube-api-access-hzswv" (OuterVolumeSpecName: "kube-api-access-hzswv") pod "e8bc5265-7b66-4876-b29d-1376c7d7d576" (UID: "e8bc5265-7b66-4876-b29d-1376c7d7d576"). InnerVolumeSpecName "kube-api-access-hzswv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:35:47.385908 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.385898 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e8bc5265-7b66-4876-b29d-1376c7d7d576" (UID: "e8bc5265-7b66-4876-b29d-1376c7d7d576"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:35:47.484548 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.484513 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:35:47.484548 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.484543 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:35:47.484548 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.484554 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bc5265-7b66-4876-b29d-1376c7d7d576-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:35:47.485190 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.484562 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bc5265-7b66-4876-b29d-1376c7d7d576-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:35:47.485190 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.484570 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:35:47.485190 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.484578 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bc5265-7b66-4876-b29d-1376c7d7d576-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:35:47.485190 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.484585 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bc5265-7b66-4876-b29d-1376c7d7d576-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:35:47.485190 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.484593 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzswv\" (UniqueName: \"kubernetes.io/projected/e8bc5265-7b66-4876-b29d-1376c7d7d576-kube-api-access-hzswv\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:35:47.640064 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.639974 2556 generic.go:356] "Generic (PLEG): container finished" podID="e8bc5265-7b66-4876-b29d-1376c7d7d576" containerID="42586f5fa80d0e6f0d231de63e58458cafb8b0a899eaf0ca89f5ae05cef8018f" exitCode=0 Sep 14 11:35:47.640064 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.640032 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" Sep 14 11:35:47.640579 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.640056 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" event={"ID":"e8bc5265-7b66-4876-b29d-1376c7d7d576","Type":"ContainerDied","Data":"42586f5fa80d0e6f0d231de63e58458cafb8b0a899eaf0ca89f5ae05cef8018f"} Sep 14 11:35:47.640579 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.640132 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9" event={"ID":"e8bc5265-7b66-4876-b29d-1376c7d7d576","Type":"ContainerDied","Data":"d54fa044b7b46fd8d171857a4a1d312b95c8f712b644352347ad49d295b0637c"} Sep 14 11:35:47.640579 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.640163 2556 scope.go:117] "RemoveContainer" containerID="42586f5fa80d0e6f0d231de63e58458cafb8b0a899eaf0ca89f5ae05cef8018f" Sep 14 11:35:47.641729 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.641706 2556 generic.go:356] "Generic (PLEG): container finished" podID="0a339534-7e42-4f4d-b4d5-52e81d594cfc" containerID="a800e48dc3c460ead0f71d3207eccae028d760ee93747e2c4ef9a1c990c53be6" exitCode=0 Sep 14 11:35:47.641827 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.641748 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-df3763-8c2gp" event={"ID":"0a339534-7e42-4f4d-b4d5-52e81d594cfc","Type":"ContainerDied","Data":"a800e48dc3c460ead0f71d3207eccae028d760ee93747e2c4ef9a1c990c53be6"} Sep 14 11:35:47.649829 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.649813 2556 scope.go:117] "RemoveContainer" containerID="05afe2f9644373ba9cc6cd6e50c9f593a81782c26812f7a2d9b5f4b4a519c2e7" Sep 14 11:35:47.657538 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.657521 2556 scope.go:117] "RemoveContainer" containerID="42586f5fa80d0e6f0d231de63e58458cafb8b0a899eaf0ca89f5ae05cef8018f" Sep 14 11:35:47.657796 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:35:47.657777 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42586f5fa80d0e6f0d231de63e58458cafb8b0a899eaf0ca89f5ae05cef8018f\": container with ID starting with 42586f5fa80d0e6f0d231de63e58458cafb8b0a899eaf0ca89f5ae05cef8018f not found: ID does not exist" containerID="42586f5fa80d0e6f0d231de63e58458cafb8b0a899eaf0ca89f5ae05cef8018f" Sep 14 11:35:47.657831 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.657807 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42586f5fa80d0e6f0d231de63e58458cafb8b0a899eaf0ca89f5ae05cef8018f"} err="failed to get container status \"42586f5fa80d0e6f0d231de63e58458cafb8b0a899eaf0ca89f5ae05cef8018f\": rpc error: code = NotFound desc = could not find container \"42586f5fa80d0e6f0d231de63e58458cafb8b0a899eaf0ca89f5ae05cef8018f\": container with ID starting with 42586f5fa80d0e6f0d231de63e58458cafb8b0a899eaf0ca89f5ae05cef8018f not found: ID does not exist" Sep 14 11:35:47.657831 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.657825 2556 scope.go:117] "RemoveContainer" containerID="05afe2f9644373ba9cc6cd6e50c9f593a81782c26812f7a2d9b5f4b4a519c2e7" Sep 14 11:35:47.658096 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:35:47.658070 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05afe2f9644373ba9cc6cd6e50c9f593a81782c26812f7a2d9b5f4b4a519c2e7\": container with ID starting with 05afe2f9644373ba9cc6cd6e50c9f593a81782c26812f7a2d9b5f4b4a519c2e7 not found: ID does not exist" containerID="05afe2f9644373ba9cc6cd6e50c9f593a81782c26812f7a2d9b5f4b4a519c2e7" Sep 14 11:35:47.658149 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.658102 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05afe2f9644373ba9cc6cd6e50c9f593a81782c26812f7a2d9b5f4b4a519c2e7"} err="failed to get container status \"05afe2f9644373ba9cc6cd6e50c9f593a81782c26812f7a2d9b5f4b4a519c2e7\": rpc error: code = NotFound desc = could not find container \"05afe2f9644373ba9cc6cd6e50c9f593a81782c26812f7a2d9b5f4b4a519c2e7\": container with ID starting with 05afe2f9644373ba9cc6cd6e50c9f593a81782c26812f7a2d9b5f4b4a519c2e7 not found: ID does not exist" Sep 14 11:35:47.675829 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.675802 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9"] Sep 14 11:35:47.678942 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:47.678916 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1ec353-openshift-default-b549d7d64-dsdx9"] Sep 14 11:35:48.382531 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:48.382491 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8bc5265-7b66-4876-b29d-1376c7d7d576" path="/var/lib/kubelet/pods/e8bc5265-7b66-4876-b29d-1376c7d7d576/volumes" Sep 14 11:35:48.777490 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:48.777468 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-df3763-8c2gp" Sep 14 11:35:48.897869 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:48.897839 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a339534-7e42-4f4d-b4d5-52e81d594cfc-maas-api-service-ca\") pod \"0a339534-7e42-4f4d-b4d5-52e81d594cfc\" (UID: \"0a339534-7e42-4f4d-b4d5-52e81d594cfc\") " Sep 14 11:35:48.898293 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:48.898264 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a339534-7e42-4f4d-b4d5-52e81d594cfc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0a339534-7e42-4f4d-b4d5-52e81d594cfc" (UID: "0a339534-7e42-4f4d-b4d5-52e81d594cfc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:35:48.999452 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:48.999371 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a339534-7e42-4f4d-b4d5-52e81d594cfc-maas-api-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:35:49.655347 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:49.655321 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-df3763-8c2gp" Sep 14 11:35:49.655347 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:49.655331 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-df3763-8c2gp" event={"ID":"0a339534-7e42-4f4d-b4d5-52e81d594cfc","Type":"ContainerDied","Data":"a7a3cbcfb2518ff245539baf1a90698f5270f1d6f98e286d3915382231cb90a6"} Sep 14 11:35:49.655571 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:35:49.655381 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7a3cbcfb2518ff245539baf1a90698f5270f1d6f98e286d3915382231cb90a6" Sep 14 11:36:01.088681 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.088633 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf"] Sep 14 11:36:01.089139 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.088909 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf" podUID="3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58" containerName="payload-processing" containerID="cri-o://9ea43572efb56db0c49e9fff922e22fec1a096dcd7fe8f5fcee89597a2825ecd" gracePeriod=30 Sep 14 11:36:01.091859 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.091825 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr"] Sep 14 11:36:01.092104 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.092035 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr" podUID="365bafbd-8258-4168-a55d-0d6a4667e47a" containerName="payload-pre-processing" containerID="cri-o://e2c2616d12bbc7ddbfdffd454d0af44190f36ffca03fd28289ad33b0dc8ad6ad" gracePeriod=30 Sep 14 11:36:01.298857 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.298806 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-df3763-8c2gp"] Sep 14 11:36:01.304133 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.304104 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-df3763-8c2gp"] Sep 14 11:36:01.384067 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.384039 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf" Sep 14 11:36:01.387757 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.387732 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr" Sep 14 11:36:01.520201 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.520175 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58-plugins-config-volume\") pod \"3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58\" (UID: \"3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58\") " Sep 14 11:36:01.520397 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.520218 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/365bafbd-8258-4168-a55d-0d6a4667e47a-plugins-config-volume\") pod \"365bafbd-8258-4168-a55d-0d6a4667e47a\" (UID: \"365bafbd-8258-4168-a55d-0d6a4667e47a\") " Sep 14 11:36:01.520397 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.520244 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjzn6\" (UniqueName: \"kubernetes.io/projected/365bafbd-8258-4168-a55d-0d6a4667e47a-kube-api-access-cjzn6\") pod \"365bafbd-8258-4168-a55d-0d6a4667e47a\" (UID: \"365bafbd-8258-4168-a55d-0d6a4667e47a\") " Sep 14 11:36:01.520397 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.520308 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjgfv\" (UniqueName: \"kubernetes.io/projected/3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58-kube-api-access-zjgfv\") pod \"3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58\" (UID: \"3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58\") " Sep 14 11:36:01.520607 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.520580 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365bafbd-8258-4168-a55d-0d6a4667e47a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "365bafbd-8258-4168-a55d-0d6a4667e47a" (UID: "365bafbd-8258-4168-a55d-0d6a4667e47a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:36:01.520667 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.520605 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58" (UID: "3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:36:01.522387 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.522367 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58-kube-api-access-zjgfv" (OuterVolumeSpecName: "kube-api-access-zjgfv") pod "3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58" (UID: "3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58"). InnerVolumeSpecName "kube-api-access-zjgfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:36:01.522463 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.522384 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365bafbd-8258-4168-a55d-0d6a4667e47a-kube-api-access-cjzn6" (OuterVolumeSpecName: "kube-api-access-cjzn6") pod "365bafbd-8258-4168-a55d-0d6a4667e47a" (UID: "365bafbd-8258-4168-a55d-0d6a4667e47a"). InnerVolumeSpecName "kube-api-access-cjzn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:36:01.621227 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.621155 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58-plugins-config-volume\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:36:01.621227 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.621180 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/365bafbd-8258-4168-a55d-0d6a4667e47a-plugins-config-volume\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:36:01.621227 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.621190 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjzn6\" (UniqueName: \"kubernetes.io/projected/365bafbd-8258-4168-a55d-0d6a4667e47a-kube-api-access-cjzn6\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:36:01.621227 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.621199 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjgfv\" (UniqueName: \"kubernetes.io/projected/3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58-kube-api-access-zjgfv\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:36:01.714601 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.714573 2556 generic.go:356] "Generic (PLEG): container finished" podID="3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58" containerID="9ea43572efb56db0c49e9fff922e22fec1a096dcd7fe8f5fcee89597a2825ecd" exitCode=0 Sep 14 11:36:01.714783 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.714616 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf" Sep 14 11:36:01.714783 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.714659 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf" event={"ID":"3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58","Type":"ContainerDied","Data":"9ea43572efb56db0c49e9fff922e22fec1a096dcd7fe8f5fcee89597a2825ecd"} Sep 14 11:36:01.714783 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.714717 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf" event={"ID":"3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58","Type":"ContainerDied","Data":"6d0fc536e336e07af246fc0b2b5bca3c5b5c8af67c18b0681db8835fe51c201e"} Sep 14 11:36:01.714783 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.714747 2556 scope.go:117] "RemoveContainer" containerID="9ea43572efb56db0c49e9fff922e22fec1a096dcd7fe8f5fcee89597a2825ecd" Sep 14 11:36:01.716243 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.716219 2556 generic.go:356] "Generic (PLEG): container finished" podID="365bafbd-8258-4168-a55d-0d6a4667e47a" containerID="e2c2616d12bbc7ddbfdffd454d0af44190f36ffca03fd28289ad33b0dc8ad6ad" exitCode=0 Sep 14 11:36:01.716368 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.716267 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr" Sep 14 11:36:01.716368 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.716284 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr" event={"ID":"365bafbd-8258-4168-a55d-0d6a4667e47a","Type":"ContainerDied","Data":"e2c2616d12bbc7ddbfdffd454d0af44190f36ffca03fd28289ad33b0dc8ad6ad"} Sep 14 11:36:01.716368 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.716335 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr" event={"ID":"365bafbd-8258-4168-a55d-0d6a4667e47a","Type":"ContainerDied","Data":"d84934c27dd23b3be236d435c01fb5ad8e076772454275c0bcab35b93c2e06ac"} Sep 14 11:36:01.723441 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.723336 2556 scope.go:117] "RemoveContainer" containerID="9ea43572efb56db0c49e9fff922e22fec1a096dcd7fe8f5fcee89597a2825ecd" Sep 14 11:36:01.723663 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:36:01.723635 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea43572efb56db0c49e9fff922e22fec1a096dcd7fe8f5fcee89597a2825ecd\": container with ID starting with 9ea43572efb56db0c49e9fff922e22fec1a096dcd7fe8f5fcee89597a2825ecd not found: ID does not exist" containerID="9ea43572efb56db0c49e9fff922e22fec1a096dcd7fe8f5fcee89597a2825ecd" Sep 14 11:36:01.723812 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.723788 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea43572efb56db0c49e9fff922e22fec1a096dcd7fe8f5fcee89597a2825ecd"} err="failed to get container status \"9ea43572efb56db0c49e9fff922e22fec1a096dcd7fe8f5fcee89597a2825ecd\": rpc error: code = NotFound desc = could not find container \"9ea43572efb56db0c49e9fff922e22fec1a096dcd7fe8f5fcee89597a2825ecd\": container with ID starting with 9ea43572efb56db0c49e9fff922e22fec1a096dcd7fe8f5fcee89597a2825ecd not found: ID does not exist" Sep 14 11:36:01.723950 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.723908 2556 scope.go:117] "RemoveContainer" containerID="e2c2616d12bbc7ddbfdffd454d0af44190f36ffca03fd28289ad33b0dc8ad6ad" Sep 14 11:36:01.733559 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.733544 2556 scope.go:117] "RemoveContainer" containerID="e2c2616d12bbc7ddbfdffd454d0af44190f36ffca03fd28289ad33b0dc8ad6ad" Sep 14 11:36:01.733831 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:36:01.733816 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2c2616d12bbc7ddbfdffd454d0af44190f36ffca03fd28289ad33b0dc8ad6ad\": container with ID starting with e2c2616d12bbc7ddbfdffd454d0af44190f36ffca03fd28289ad33b0dc8ad6ad not found: ID does not exist" containerID="e2c2616d12bbc7ddbfdffd454d0af44190f36ffca03fd28289ad33b0dc8ad6ad" Sep 14 11:36:01.733890 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.733840 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2c2616d12bbc7ddbfdffd454d0af44190f36ffca03fd28289ad33b0dc8ad6ad"} err="failed to get container status \"e2c2616d12bbc7ddbfdffd454d0af44190f36ffca03fd28289ad33b0dc8ad6ad\": rpc error: code = NotFound desc = could not find container \"e2c2616d12bbc7ddbfdffd454d0af44190f36ffca03fd28289ad33b0dc8ad6ad\": container with ID starting with e2c2616d12bbc7ddbfdffd454d0af44190f36ffca03fd28289ad33b0dc8ad6ad not found: ID does not exist" Sep 14 11:36:01.741767 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.741735 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr"] Sep 14 11:36:01.743426 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.743408 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-df3763-766755848b-j76gr"] Sep 14 11:36:01.753336 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.753313 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf"] Sep 14 11:36:01.757119 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:01.757084 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-df3763-685f7ffbf9-qfrpf"] Sep 14 11:36:02.382387 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:02.382353 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a339534-7e42-4f4d-b4d5-52e81d594cfc" path="/var/lib/kubelet/pods/0a339534-7e42-4f4d-b4d5-52e81d594cfc/volumes" Sep 14 11:36:02.382879 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:02.382860 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="365bafbd-8258-4168-a55d-0d6a4667e47a" path="/var/lib/kubelet/pods/365bafbd-8258-4168-a55d-0d6a4667e47a/volumes" Sep 14 11:36:02.383221 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:02.383208 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58" path="/var/lib/kubelet/pods/3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58/volumes" Sep 14 11:36:42.832903 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:42.832821 2556 scope.go:117] "RemoveContainer" containerID="b97ff7fbcdaf06fc0ff502661b6098b3e00424b69cd5eb09eebb9c3f2f4d0ac5" Sep 14 11:36:42.840728 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:42.840709 2556 scope.go:117] "RemoveContainer" containerID="68edfaf0c3af5f81c17ea36c3dfa51a7547f5803c65db12c019473e2967b5559" Sep 14 11:36:42.847677 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:42.847658 2556 scope.go:117] "RemoveContainer" containerID="5e95a351c5cfd850f43f2556f5e76227937d51a9f561979237088814c3193aea" Sep 14 11:36:49.650507 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:49.650467 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr"] Sep 14 11:36:49.650896 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:49.650714 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" podUID="4052cd69-fa63-4d4c-a598-91fa8588ef5d" containerName="istio-proxy" containerID="cri-o://19e40a3c5454bf1f2a65708efe5160141e14022888874194fd1e06e2c02d8b88" gracePeriod=30 Sep 14 11:36:49.995504 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:49.995419 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d0f937-mnvtw"] Sep 14 11:36:49.995732 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:49.995720 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8bc5265-7b66-4876-b29d-1376c7d7d576" containerName="istio-proxy" Sep 14 11:36:49.995798 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:49.995733 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8bc5265-7b66-4876-b29d-1376c7d7d576" containerName="istio-proxy" Sep 14 11:36:49.995798 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:49.995744 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58" containerName="payload-processing" Sep 14 11:36:49.995798 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:49.995750 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58" containerName="payload-processing" Sep 14 11:36:49.995798 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:49.995760 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a339534-7e42-4f4d-b4d5-52e81d594cfc" containerName="revoke-keys" Sep 14 11:36:49.995798 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:49.995765 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a339534-7e42-4f4d-b4d5-52e81d594cfc" containerName="revoke-keys" Sep 14 11:36:49.995798 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:49.995773 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="365bafbd-8258-4168-a55d-0d6a4667e47a" containerName="payload-pre-processing" Sep 14 11:36:49.995798 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:49.995780 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="365bafbd-8258-4168-a55d-0d6a4667e47a" containerName="payload-pre-processing" Sep 14 11:36:49.995798 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:49.995789 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8bc5265-7b66-4876-b29d-1376c7d7d576" containerName="istio-proxy" Sep 14 11:36:49.995798 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:49.995794 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8bc5265-7b66-4876-b29d-1376c7d7d576" containerName="istio-proxy" Sep 14 11:36:49.996248 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:49.995843 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8bc5265-7b66-4876-b29d-1376c7d7d576" containerName="istio-proxy" Sep 14 11:36:49.996248 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:49.995850 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="365bafbd-8258-4168-a55d-0d6a4667e47a" containerName="payload-pre-processing" Sep 14 11:36:49.996248 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:49.995856 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ba63976-95cb-4b46-8bb0-4dbc3f2bfa58" containerName="payload-processing" Sep 14 11:36:49.996248 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:49.995862 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8bc5265-7b66-4876-b29d-1376c7d7d576" containerName="istio-proxy" Sep 14 11:36:49.996248 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:49.995869 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a339534-7e42-4f4d-b4d5-52e81d594cfc" containerName="revoke-keys" Sep 14 11:36:49.998537 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:49.998519 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d0f937-mnvtw" Sep 14 11:36:50.000751 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:50.000729 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 11:36:50.000891 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:50.000792 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f98vt\"" Sep 14 11:36:50.008482 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:50.008460 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d0f937-mnvtw"] Sep 14 11:36:50.120422 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:50.120396 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd26d85-359f-443e-a967-779aaa608408-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d0f937-mnvtw\" (UID: \"0dd26d85-359f-443e-a967-779aaa608408\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d0f937-mnvtw" Sep 14 11:36:50.221711 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:50.221679 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd26d85-359f-443e-a967-779aaa608408-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d0f937-mnvtw\" (UID: \"0dd26d85-359f-443e-a967-779aaa608408\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d0f937-mnvtw" Sep 14 11:36:50.222298 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:50.222282 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd26d85-359f-443e-a967-779aaa608408-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d0f937-mnvtw\" (UID: \"0dd26d85-359f-443e-a967-779aaa608408\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d0f937-mnvtw" Sep 14 11:36:50.308656 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:50.308585 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d0f937-mnvtw" Sep 14 11:36:50.578619 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:50.578598 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d0f937-mnvtw"] Sep 14 11:36:50.580647 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:36:50.580622 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd26d85_359f_443e_a967_779aaa608408.slice/crio-21a780b47b47f58b250aea4555a3acc35c80563b0131a4459cc72f029b869356 WatchSource:0}: Error finding container 21a780b47b47f58b250aea4555a3acc35c80563b0131a4459cc72f029b869356: Status 404 returned error can't find the container with id 21a780b47b47f58b250aea4555a3acc35c80563b0131a4459cc72f029b869356 Sep 14 11:36:50.965081 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:50.965053 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d0f937-mnvtw" event={"ID":"0dd26d85-359f-443e-a967-779aaa608408","Type":"ContainerStarted","Data":"4c8f99f42eaf969bb12b78000136d9904a589b58854634011856e4e36f821c2f"} Sep 14 11:36:50.965435 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:50.965124 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d0f937-mnvtw" event={"ID":"0dd26d85-359f-443e-a967-779aaa608408","Type":"ContainerStarted","Data":"21a780b47b47f58b250aea4555a3acc35c80563b0131a4459cc72f029b869356"} Sep 14 11:36:50.982484 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:50.982431 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d0f937-mnvtw" podStartSLOduration=1.982414428 podStartE2EDuration="1.982414428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:36:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:36:50.980370778 +0000 UTC m=+2169.043773688" watchObservedRunningTime="2026-09-14 11:36:50.982414428 +0000 UTC m=+2169.045817339" Sep 14 11:36:51.971591 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:51.971555 2556 generic.go:356] "Generic (PLEG): container finished" podID="0dd26d85-359f-443e-a967-779aaa608408" containerID="4c8f99f42eaf969bb12b78000136d9904a589b58854634011856e4e36f821c2f" exitCode=0 Sep 14 11:36:51.971948 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:51.971639 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d0f937-mnvtw" event={"ID":"0dd26d85-359f-443e-a967-779aaa608408","Type":"ContainerDied","Data":"4c8f99f42eaf969bb12b78000136d9904a589b58854634011856e4e36f821c2f"} Sep 14 11:36:53.100163 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:53.100137 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d0f937-mnvtw" Sep 14 11:36:53.247691 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:53.247623 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd26d85-359f-443e-a967-779aaa608408-maas-api-service-ca\") pod \"0dd26d85-359f-443e-a967-779aaa608408\" (UID: \"0dd26d85-359f-443e-a967-779aaa608408\") " Sep 14 11:36:53.247956 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:53.247935 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd26d85-359f-443e-a967-779aaa608408-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0dd26d85-359f-443e-a967-779aaa608408" (UID: "0dd26d85-359f-443e-a967-779aaa608408"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:36:53.348743 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:53.348713 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd26d85-359f-443e-a967-779aaa608408-maas-api-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:36:53.982278 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:53.982251 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d0f937-mnvtw" Sep 14 11:36:53.982479 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:53.982244 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d0f937-mnvtw" event={"ID":"0dd26d85-359f-443e-a967-779aaa608408","Type":"ContainerDied","Data":"21a780b47b47f58b250aea4555a3acc35c80563b0131a4459cc72f029b869356"} Sep 14 11:36:53.982479 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:53.982412 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21a780b47b47f58b250aea4555a3acc35c80563b0131a4459cc72f029b869356" Sep 14 11:36:54.895927 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:54.895903 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:36:54.988697 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:54.988633 2556 generic.go:356] "Generic (PLEG): container finished" podID="4052cd69-fa63-4d4c-a598-91fa8588ef5d" containerID="19e40a3c5454bf1f2a65708efe5160141e14022888874194fd1e06e2c02d8b88" exitCode=0 Sep 14 11:36:54.988697 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:54.988677 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" Sep 14 11:36:54.988848 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:54.988713 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" event={"ID":"4052cd69-fa63-4d4c-a598-91fa8588ef5d","Type":"ContainerDied","Data":"19e40a3c5454bf1f2a65708efe5160141e14022888874194fd1e06e2c02d8b88"} Sep 14 11:36:54.988848 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:54.988768 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr" event={"ID":"4052cd69-fa63-4d4c-a598-91fa8588ef5d","Type":"ContainerDied","Data":"a0b9a8dce16a5a78d7d1d229fcec42ba31e9f5b3a480aa1441c0a9ed06d90735"} Sep 14 11:36:54.988848 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:54.988796 2556 scope.go:117] "RemoveContainer" containerID="19e40a3c5454bf1f2a65708efe5160141e14022888874194fd1e06e2c02d8b88" Sep 14 11:36:54.996738 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:54.996719 2556 scope.go:117] "RemoveContainer" containerID="0dc12bd05847cd9b5108046c7a0659dd86feebefe05ea73dd758372e93f0c6f7" Sep 14 11:36:55.003797 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.003779 2556 scope.go:117] "RemoveContainer" containerID="19e40a3c5454bf1f2a65708efe5160141e14022888874194fd1e06e2c02d8b88" Sep 14 11:36:55.004058 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:36:55.004039 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19e40a3c5454bf1f2a65708efe5160141e14022888874194fd1e06e2c02d8b88\": container with ID starting with 19e40a3c5454bf1f2a65708efe5160141e14022888874194fd1e06e2c02d8b88 not found: ID does not exist" containerID="19e40a3c5454bf1f2a65708efe5160141e14022888874194fd1e06e2c02d8b88" Sep 14 11:36:55.004133 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.004067 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e40a3c5454bf1f2a65708efe5160141e14022888874194fd1e06e2c02d8b88"} err="failed to get container status \"19e40a3c5454bf1f2a65708efe5160141e14022888874194fd1e06e2c02d8b88\": rpc error: code = NotFound desc = could not find container \"19e40a3c5454bf1f2a65708efe5160141e14022888874194fd1e06e2c02d8b88\": container with ID starting with 19e40a3c5454bf1f2a65708efe5160141e14022888874194fd1e06e2c02d8b88 not found: ID does not exist" Sep 14 11:36:55.004133 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.004105 2556 scope.go:117] "RemoveContainer" containerID="0dc12bd05847cd9b5108046c7a0659dd86feebefe05ea73dd758372e93f0c6f7" Sep 14 11:36:55.004358 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:36:55.004340 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dc12bd05847cd9b5108046c7a0659dd86feebefe05ea73dd758372e93f0c6f7\": container with ID starting with 0dc12bd05847cd9b5108046c7a0659dd86feebefe05ea73dd758372e93f0c6f7 not found: ID does not exist" containerID="0dc12bd05847cd9b5108046c7a0659dd86feebefe05ea73dd758372e93f0c6f7" Sep 14 11:36:55.004413 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.004365 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc12bd05847cd9b5108046c7a0659dd86feebefe05ea73dd758372e93f0c6f7"} err="failed to get container status \"0dc12bd05847cd9b5108046c7a0659dd86feebefe05ea73dd758372e93f0c6f7\": rpc error: code = NotFound desc = could not find container \"0dc12bd05847cd9b5108046c7a0659dd86feebefe05ea73dd758372e93f0c6f7\": container with ID starting with 0dc12bd05847cd9b5108046c7a0659dd86feebefe05ea73dd758372e93f0c6f7 not found: ID does not exist" Sep 14 11:36:55.061574 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.061554 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istio-data\") pod \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " Sep 14 11:36:55.061658 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.061584 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istio-envoy\") pod \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " Sep 14 11:36:55.061658 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.061608 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-credential-socket\") pod \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " Sep 14 11:36:55.061739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.061663 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istio-podinfo\") pod \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " Sep 14 11:36:55.061739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.061689 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nt68z\" (UniqueName: \"kubernetes.io/projected/4052cd69-fa63-4d4c-a598-91fa8588ef5d-kube-api-access-nt68z\") pod \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " Sep 14 11:36:55.061739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.061707 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istio-token\") pod \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " Sep 14 11:36:55.061739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.061721 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-workload-socket\") pod \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " Sep 14 11:36:55.061930 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.061762 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-workload-certs\") pod \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " Sep 14 11:36:55.061930 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.061796 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istiod-ca-cert\") pod \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\" (UID: \"4052cd69-fa63-4d4c-a598-91fa8588ef5d\") " Sep 14 11:36:55.062037 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.062011 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4052cd69-fa63-4d4c-a598-91fa8588ef5d" (UID: "4052cd69-fa63-4d4c-a598-91fa8588ef5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:36:55.062115 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.062027 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4052cd69-fa63-4d4c-a598-91fa8588ef5d" (UID: "4052cd69-fa63-4d4c-a598-91fa8588ef5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:36:55.062180 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.062139 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:36:55.062180 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.062160 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:36:55.062286 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.062265 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istio-data" (OuterVolumeSpecName: "istio-data") pod "4052cd69-fa63-4d4c-a598-91fa8588ef5d" (UID: "4052cd69-fa63-4d4c-a598-91fa8588ef5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:36:55.062350 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.062335 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4052cd69-fa63-4d4c-a598-91fa8588ef5d" (UID: "4052cd69-fa63-4d4c-a598-91fa8588ef5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:36:55.062395 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.062347 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4052cd69-fa63-4d4c-a598-91fa8588ef5d" (UID: "4052cd69-fa63-4d4c-a598-91fa8588ef5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:36:55.063690 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.063666 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4052cd69-fa63-4d4c-a598-91fa8588ef5d-kube-api-access-nt68z" (OuterVolumeSpecName: "kube-api-access-nt68z") pod "4052cd69-fa63-4d4c-a598-91fa8588ef5d" (UID: "4052cd69-fa63-4d4c-a598-91fa8588ef5d"). InnerVolumeSpecName "kube-api-access-nt68z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:36:55.063801 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.063774 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istio-token" (OuterVolumeSpecName: "istio-token") pod "4052cd69-fa63-4d4c-a598-91fa8588ef5d" (UID: "4052cd69-fa63-4d4c-a598-91fa8588ef5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:36:55.063940 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.063920 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4052cd69-fa63-4d4c-a598-91fa8588ef5d" (UID: "4052cd69-fa63-4d4c-a598-91fa8588ef5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 11:36:55.063987 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.063924 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4052cd69-fa63-4d4c-a598-91fa8588ef5d" (UID: "4052cd69-fa63-4d4c-a598-91fa8588ef5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:36:55.162991 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.162966 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:36:55.162991 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.162989 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:36:55.163162 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.163003 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:36:55.163162 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.163014 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:36:55.163162 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.163024 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:36:55.163162 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.163035 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4052cd69-fa63-4d4c-a598-91fa8588ef5d-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:36:55.163162 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.163046 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt68z\" (UniqueName: \"kubernetes.io/projected/4052cd69-fa63-4d4c-a598-91fa8588ef5d-kube-api-access-nt68z\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:36:55.311177 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.311149 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr"] Sep 14 11:36:55.314965 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:55.314941 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-df3763-openshift-default-84b7594fd4-4tnmr"] Sep 14 11:36:56.382449 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:36:56.382416 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4052cd69-fa63-4d4c-a598-91fa8588ef5d" path="/var/lib/kubelet/pods/4052cd69-fa63-4d4c-a598-91fa8588ef5d/volumes" Sep 14 11:37:00.624178 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:00.624142 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42"] Sep 14 11:37:00.624619 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:00.624445 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42" podUID="69443040-eb9a-4a5e-8783-40b17474557b" containerName="payload-processing" containerID="cri-o://1bca724dc9b743894eec37dfcfef976de354bbf89eb8a1f10b82f270213bbec9" gracePeriod=30 Sep 14 11:37:00.777488 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:00.777452 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d0f937-mnvtw"] Sep 14 11:37:00.779006 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:00.778982 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d0f937-mnvtw"] Sep 14 11:37:00.869774 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:00.869741 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42" Sep 14 11:37:01.012168 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:01.012064 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69443040-eb9a-4a5e-8783-40b17474557b-plugins-config-volume\") pod \"69443040-eb9a-4a5e-8783-40b17474557b\" (UID: \"69443040-eb9a-4a5e-8783-40b17474557b\") " Sep 14 11:37:01.012168 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:01.012148 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jknnk\" (UniqueName: \"kubernetes.io/projected/69443040-eb9a-4a5e-8783-40b17474557b-kube-api-access-jknnk\") pod \"69443040-eb9a-4a5e-8783-40b17474557b\" (UID: \"69443040-eb9a-4a5e-8783-40b17474557b\") " Sep 14 11:37:01.012440 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:01.012414 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69443040-eb9a-4a5e-8783-40b17474557b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "69443040-eb9a-4a5e-8783-40b17474557b" (UID: "69443040-eb9a-4a5e-8783-40b17474557b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:37:01.014110 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:01.014068 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69443040-eb9a-4a5e-8783-40b17474557b-kube-api-access-jknnk" (OuterVolumeSpecName: "kube-api-access-jknnk") pod "69443040-eb9a-4a5e-8783-40b17474557b" (UID: "69443040-eb9a-4a5e-8783-40b17474557b"). InnerVolumeSpecName "kube-api-access-jknnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:37:01.021256 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:01.021228 2556 generic.go:356] "Generic (PLEG): container finished" podID="69443040-eb9a-4a5e-8783-40b17474557b" containerID="1bca724dc9b743894eec37dfcfef976de354bbf89eb8a1f10b82f270213bbec9" exitCode=0 Sep 14 11:37:01.021351 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:01.021274 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42" Sep 14 11:37:01.021351 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:01.021319 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42" event={"ID":"69443040-eb9a-4a5e-8783-40b17474557b","Type":"ContainerDied","Data":"1bca724dc9b743894eec37dfcfef976de354bbf89eb8a1f10b82f270213bbec9"} Sep 14 11:37:01.021427 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:01.021384 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42" event={"ID":"69443040-eb9a-4a5e-8783-40b17474557b","Type":"ContainerDied","Data":"f8b287c465967dac12eb709a703f855442febe0da1d26b82a5bd8216a60e645c"} Sep 14 11:37:01.021427 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:01.021417 2556 scope.go:117] "RemoveContainer" containerID="1bca724dc9b743894eec37dfcfef976de354bbf89eb8a1f10b82f270213bbec9" Sep 14 11:37:01.030786 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:01.030769 2556 scope.go:117] "RemoveContainer" containerID="1bca724dc9b743894eec37dfcfef976de354bbf89eb8a1f10b82f270213bbec9" Sep 14 11:37:01.031064 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:37:01.031047 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bca724dc9b743894eec37dfcfef976de354bbf89eb8a1f10b82f270213bbec9\": container with ID starting with 1bca724dc9b743894eec37dfcfef976de354bbf89eb8a1f10b82f270213bbec9 not found: ID does not exist" containerID="1bca724dc9b743894eec37dfcfef976de354bbf89eb8a1f10b82f270213bbec9" Sep 14 11:37:01.031143 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:01.031075 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bca724dc9b743894eec37dfcfef976de354bbf89eb8a1f10b82f270213bbec9"} err="failed to get container status \"1bca724dc9b743894eec37dfcfef976de354bbf89eb8a1f10b82f270213bbec9\": rpc error: code = NotFound desc = could not find container \"1bca724dc9b743894eec37dfcfef976de354bbf89eb8a1f10b82f270213bbec9\": container with ID starting with 1bca724dc9b743894eec37dfcfef976de354bbf89eb8a1f10b82f270213bbec9 not found: ID does not exist" Sep 14 11:37:01.041999 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:01.041977 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42"] Sep 14 11:37:01.046191 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:01.046170 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d0f937-58b8f4796f-8vv42"] Sep 14 11:37:01.113367 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:01.113343 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69443040-eb9a-4a5e-8783-40b17474557b-plugins-config-volume\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:37:01.113367 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:01.113365 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jknnk\" (UniqueName: \"kubernetes.io/projected/69443040-eb9a-4a5e-8783-40b17474557b-kube-api-access-jknnk\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:37:02.382192 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:02.382156 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dd26d85-359f-443e-a967-779aaa608408" path="/var/lib/kubelet/pods/0dd26d85-359f-443e-a967-779aaa608408/volumes" Sep 14 11:37:02.382555 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:02.382525 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69443040-eb9a-4a5e-8783-40b17474557b" path="/var/lib/kubelet/pods/69443040-eb9a-4a5e-8783-40b17474557b/volumes" Sep 14 11:37:07.590832 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:07.590800 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-55c025-h4cjj"] Sep 14 11:37:07.591232 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:07.591117 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4052cd69-fa63-4d4c-a598-91fa8588ef5d" containerName="istio-proxy" Sep 14 11:37:07.591232 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:07.591134 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4052cd69-fa63-4d4c-a598-91fa8588ef5d" containerName="istio-proxy" Sep 14 11:37:07.591232 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:07.591146 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4052cd69-fa63-4d4c-a598-91fa8588ef5d" containerName="istio-proxy" Sep 14 11:37:07.591232 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:07.591152 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4052cd69-fa63-4d4c-a598-91fa8588ef5d" containerName="istio-proxy" Sep 14 11:37:07.591232 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:07.591168 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69443040-eb9a-4a5e-8783-40b17474557b" containerName="payload-processing" Sep 14 11:37:07.591232 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:07.591174 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="69443040-eb9a-4a5e-8783-40b17474557b" containerName="payload-processing" Sep 14 11:37:07.591232 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:07.591186 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dd26d85-359f-443e-a967-779aaa608408" containerName="revoke-keys" Sep 14 11:37:07.591232 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:07.591191 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dd26d85-359f-443e-a967-779aaa608408" containerName="revoke-keys" Sep 14 11:37:07.591472 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:07.591245 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dd26d85-359f-443e-a967-779aaa608408" containerName="revoke-keys" Sep 14 11:37:07.591472 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:07.591252 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4052cd69-fa63-4d4c-a598-91fa8588ef5d" containerName="istio-proxy" Sep 14 11:37:07.591472 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:07.591260 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4052cd69-fa63-4d4c-a598-91fa8588ef5d" containerName="istio-proxy" Sep 14 11:37:07.591472 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:07.591266 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="69443040-eb9a-4a5e-8783-40b17474557b" containerName="payload-processing" Sep 14 11:37:07.595345 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:07.595328 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-55c025-h4cjj" Sep 14 11:37:07.597791 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:07.597771 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 11:37:07.597891 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:07.597810 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f98vt\"" Sep 14 11:37:07.602650 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:07.602315 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-55c025-h4cjj"] Sep 14 11:37:07.657237 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:07.657215 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a01f6a-13fd-450f-9c58-6e79d67b0f59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-55c025-h4cjj\" (UID: \"e7a01f6a-13fd-450f-9c58-6e79d67b0f59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-55c025-h4cjj" Sep 14 11:37:07.758395 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:07.758369 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a01f6a-13fd-450f-9c58-6e79d67b0f59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-55c025-h4cjj\" (UID: \"e7a01f6a-13fd-450f-9c58-6e79d67b0f59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-55c025-h4cjj" Sep 14 11:37:07.758968 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:07.758949 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a01f6a-13fd-450f-9c58-6e79d67b0f59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-55c025-h4cjj\" (UID: \"e7a01f6a-13fd-450f-9c58-6e79d67b0f59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-55c025-h4cjj" Sep 14 11:37:07.906276 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:07.906244 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-55c025-h4cjj" Sep 14 11:37:08.178048 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:08.178022 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-55c025-h4cjj"] Sep 14 11:37:08.180175 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:37:08.180133 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a01f6a_13fd_450f_9c58_6e79d67b0f59.slice/crio-53b1f7678093c5326822e44d30e893918015e9b717e38cc86bf9751b08e24b1e WatchSource:0}: Error finding container 53b1f7678093c5326822e44d30e893918015e9b717e38cc86bf9751b08e24b1e: Status 404 returned error can't find the container with id 53b1f7678093c5326822e44d30e893918015e9b717e38cc86bf9751b08e24b1e Sep 14 11:37:09.060002 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:09.059967 2556 generic.go:356] "Generic (PLEG): container finished" podID="e7a01f6a-13fd-450f-9c58-6e79d67b0f59" containerID="6e9ac40727d324901d427a922f16c1739d42d272bfb338c2f1bc83568bab7f01" exitCode=0 Sep 14 11:37:09.060343 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:09.060059 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-55c025-h4cjj" event={"ID":"e7a01f6a-13fd-450f-9c58-6e79d67b0f59","Type":"ContainerDied","Data":"6e9ac40727d324901d427a922f16c1739d42d272bfb338c2f1bc83568bab7f01"} Sep 14 11:37:09.060343 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:09.060173 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-55c025-h4cjj" event={"ID":"e7a01f6a-13fd-450f-9c58-6e79d67b0f59","Type":"ContainerStarted","Data":"53b1f7678093c5326822e44d30e893918015e9b717e38cc86bf9751b08e24b1e"} Sep 14 11:37:10.191067 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:10.191039 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-55c025-h4cjj" Sep 14 11:37:10.282342 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:10.282308 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a01f6a-13fd-450f-9c58-6e79d67b0f59-maas-api-service-ca\") pod \"e7a01f6a-13fd-450f-9c58-6e79d67b0f59\" (UID: \"e7a01f6a-13fd-450f-9c58-6e79d67b0f59\") " Sep 14 11:37:10.282615 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:10.282593 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7a01f6a-13fd-450f-9c58-6e79d67b0f59-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e7a01f6a-13fd-450f-9c58-6e79d67b0f59" (UID: "e7a01f6a-13fd-450f-9c58-6e79d67b0f59"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:37:10.382961 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:10.382930 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a01f6a-13fd-450f-9c58-6e79d67b0f59-maas-api-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:37:11.072509 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:11.072476 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-55c025-h4cjj" event={"ID":"e7a01f6a-13fd-450f-9c58-6e79d67b0f59","Type":"ContainerDied","Data":"53b1f7678093c5326822e44d30e893918015e9b717e38cc86bf9751b08e24b1e"} Sep 14 11:37:11.072509 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:11.072498 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-55c025-h4cjj" Sep 14 11:37:11.072715 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:11.072522 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53b1f7678093c5326822e44d30e893918015e9b717e38cc86bf9751b08e24b1e" Sep 14 11:37:18.096548 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:18.096511 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-55c025-29823090-bmw9b"] Sep 14 11:37:18.100645 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:18.100620 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-55c025-29823090-bmw9b"] Sep 14 11:37:18.176278 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:18.176249 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx"] Sep 14 11:37:18.176483 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:18.176446 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx" podUID="45ec7d33-3016-4f95-91a3-1fe8e039727f" containerName="payload-pre-processing" containerID="cri-o://f4ff6363117c18f12fba6157a004943ab7273146e36a67bc0c2a6a76c8ee7026" gracePeriod=30 Sep 14 11:37:18.382951 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:18.382774 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5c1f9a0-070e-4a4d-a9c8-05eaf0381b5f" path="/var/lib/kubelet/pods/e5c1f9a0-070e-4a4d-a9c8-05eaf0381b5f/volumes" Sep 14 11:37:18.391369 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:18.391344 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-55c025-h4cjj"] Sep 14 11:37:18.395807 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:18.395783 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-55c025-h4cjj"] Sep 14 11:37:18.419614 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:18.419594 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx" Sep 14 11:37:18.443842 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:18.443821 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8x52\" (UniqueName: \"kubernetes.io/projected/45ec7d33-3016-4f95-91a3-1fe8e039727f-kube-api-access-q8x52\") pod \"45ec7d33-3016-4f95-91a3-1fe8e039727f\" (UID: \"45ec7d33-3016-4f95-91a3-1fe8e039727f\") " Sep 14 11:37:18.443955 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:18.443897 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ec7d33-3016-4f95-91a3-1fe8e039727f-plugins-config-volume\") pod \"45ec7d33-3016-4f95-91a3-1fe8e039727f\" (UID: \"45ec7d33-3016-4f95-91a3-1fe8e039727f\") " Sep 14 11:37:18.444254 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:18.444230 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ec7d33-3016-4f95-91a3-1fe8e039727f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "45ec7d33-3016-4f95-91a3-1fe8e039727f" (UID: "45ec7d33-3016-4f95-91a3-1fe8e039727f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:37:18.445719 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:18.445698 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ec7d33-3016-4f95-91a3-1fe8e039727f-kube-api-access-q8x52" (OuterVolumeSpecName: "kube-api-access-q8x52") pod "45ec7d33-3016-4f95-91a3-1fe8e039727f" (UID: "45ec7d33-3016-4f95-91a3-1fe8e039727f"). InnerVolumeSpecName "kube-api-access-q8x52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:37:18.544693 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:18.544663 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ec7d33-3016-4f95-91a3-1fe8e039727f-plugins-config-volume\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:37:18.544693 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:18.544687 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8x52\" (UniqueName: \"kubernetes.io/projected/45ec7d33-3016-4f95-91a3-1fe8e039727f-kube-api-access-q8x52\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:37:19.112189 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:19.112152 2556 generic.go:356] "Generic (PLEG): container finished" podID="45ec7d33-3016-4f95-91a3-1fe8e039727f" containerID="f4ff6363117c18f12fba6157a004943ab7273146e36a67bc0c2a6a76c8ee7026" exitCode=0 Sep 14 11:37:19.112189 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:19.112194 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx" event={"ID":"45ec7d33-3016-4f95-91a3-1fe8e039727f","Type":"ContainerDied","Data":"f4ff6363117c18f12fba6157a004943ab7273146e36a67bc0c2a6a76c8ee7026"} Sep 14 11:37:19.112710 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:19.112196 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx" Sep 14 11:37:19.112710 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:19.112243 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx" event={"ID":"45ec7d33-3016-4f95-91a3-1fe8e039727f","Type":"ContainerDied","Data":"52d197d029bc3ce725eaa3b49d2ec6129e949ad14aa8b04c5ea9c3c2e9ef261e"} Sep 14 11:37:19.112710 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:19.112268 2556 scope.go:117] "RemoveContainer" containerID="f4ff6363117c18f12fba6157a004943ab7273146e36a67bc0c2a6a76c8ee7026" Sep 14 11:37:19.121104 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:19.121058 2556 scope.go:117] "RemoveContainer" containerID="f4ff6363117c18f12fba6157a004943ab7273146e36a67bc0c2a6a76c8ee7026" Sep 14 11:37:19.121443 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:37:19.121415 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4ff6363117c18f12fba6157a004943ab7273146e36a67bc0c2a6a76c8ee7026\": container with ID starting with f4ff6363117c18f12fba6157a004943ab7273146e36a67bc0c2a6a76c8ee7026 not found: ID does not exist" containerID="f4ff6363117c18f12fba6157a004943ab7273146e36a67bc0c2a6a76c8ee7026" Sep 14 11:37:19.121560 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:19.121455 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4ff6363117c18f12fba6157a004943ab7273146e36a67bc0c2a6a76c8ee7026"} err="failed to get container status \"f4ff6363117c18f12fba6157a004943ab7273146e36a67bc0c2a6a76c8ee7026\": rpc error: code = NotFound desc = could not find container \"f4ff6363117c18f12fba6157a004943ab7273146e36a67bc0c2a6a76c8ee7026\": container with ID starting with f4ff6363117c18f12fba6157a004943ab7273146e36a67bc0c2a6a76c8ee7026 not found: ID does not exist" Sep 14 11:37:19.132141 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:19.132118 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx"] Sep 14 11:37:19.135773 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:19.135752 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-55c025-67f9649d7d-x84dx"] Sep 14 11:37:20.382147 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:20.382118 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45ec7d33-3016-4f95-91a3-1fe8e039727f" path="/var/lib/kubelet/pods/45ec7d33-3016-4f95-91a3-1fe8e039727f/volumes" Sep 14 11:37:20.382588 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:20.382439 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7a01f6a-13fd-450f-9c58-6e79d67b0f59" path="/var/lib/kubelet/pods/e7a01f6a-13fd-450f-9c58-6e79d67b0f59/volumes" Sep 14 11:37:25.144698 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:25.144664 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8"] Sep 14 11:37:25.145128 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:25.144892 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" podUID="cf338544-9d39-4e9e-9668-ba204a6e2fde" containerName="istio-proxy" containerID="cri-o://b8df8986d3a73c2abfb3fc1721a11a73b8935b8dac6693bbce9dbf4c8eea9e21" gracePeriod=30 Sep 14 11:37:25.492907 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:25.492835 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3f02a5-8nmlj"] Sep 14 11:37:25.493180 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:25.493156 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45ec7d33-3016-4f95-91a3-1fe8e039727f" containerName="payload-pre-processing" Sep 14 11:37:25.493180 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:25.493181 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="45ec7d33-3016-4f95-91a3-1fe8e039727f" containerName="payload-pre-processing" Sep 14 11:37:25.493280 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:25.493196 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7a01f6a-13fd-450f-9c58-6e79d67b0f59" containerName="revoke-keys" Sep 14 11:37:25.493280 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:25.493201 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7a01f6a-13fd-450f-9c58-6e79d67b0f59" containerName="revoke-keys" Sep 14 11:37:25.493280 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:25.493267 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7a01f6a-13fd-450f-9c58-6e79d67b0f59" containerName="revoke-keys" Sep 14 11:37:25.493280 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:25.493277 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="45ec7d33-3016-4f95-91a3-1fe8e039727f" containerName="payload-pre-processing" Sep 14 11:37:25.497424 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:25.497408 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3f02a5-8nmlj" Sep 14 11:37:25.503388 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:25.500142 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f98vt\"" Sep 14 11:37:25.503388 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:25.500170 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 11:37:25.505187 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:25.505059 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3f02a5-8nmlj"] Sep 14 11:37:25.606673 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:25.606645 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86325738-3dcd-4dda-b318-bec55806e592-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-3f02a5-8nmlj\" (UID: \"86325738-3dcd-4dda-b318-bec55806e592\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3f02a5-8nmlj" Sep 14 11:37:25.707541 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:25.707507 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86325738-3dcd-4dda-b318-bec55806e592-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-3f02a5-8nmlj\" (UID: \"86325738-3dcd-4dda-b318-bec55806e592\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3f02a5-8nmlj" Sep 14 11:37:25.708121 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:25.708077 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86325738-3dcd-4dda-b318-bec55806e592-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-3f02a5-8nmlj\" (UID: \"86325738-3dcd-4dda-b318-bec55806e592\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3f02a5-8nmlj" Sep 14 11:37:25.813957 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:25.813887 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3f02a5-8nmlj" Sep 14 11:37:26.079692 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:26.079669 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3f02a5-8nmlj"] Sep 14 11:37:26.081490 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:37:26.081462 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86325738_3dcd_4dda_b318_bec55806e592.slice/crio-e1fb25572b9ee9bb4f5b05b0d9f61efb86e9727f68e057e733f52b951ba992f7 WatchSource:0}: Error finding container e1fb25572b9ee9bb4f5b05b0d9f61efb86e9727f68e057e733f52b951ba992f7: Status 404 returned error can't find the container with id e1fb25572b9ee9bb4f5b05b0d9f61efb86e9727f68e057e733f52b951ba992f7 Sep 14 11:37:26.146734 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:26.146708 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3f02a5-8nmlj" event={"ID":"86325738-3dcd-4dda-b318-bec55806e592","Type":"ContainerStarted","Data":"e1fb25572b9ee9bb4f5b05b0d9f61efb86e9727f68e057e733f52b951ba992f7"} Sep 14 11:37:27.153583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:27.153548 2556 generic.go:356] "Generic (PLEG): container finished" podID="86325738-3dcd-4dda-b318-bec55806e592" containerID="2dd39d115dc439adb2e5b7e5667cc125385541e810951e04b1ab46cc96a54b2d" exitCode=0 Sep 14 11:37:27.153942 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:27.153621 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3f02a5-8nmlj" event={"ID":"86325738-3dcd-4dda-b318-bec55806e592","Type":"ContainerDied","Data":"2dd39d115dc439adb2e5b7e5667cc125385541e810951e04b1ab46cc96a54b2d"} Sep 14 11:37:28.279691 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:28.279668 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3f02a5-8nmlj" Sep 14 11:37:28.326742 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:28.326716 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86325738-3dcd-4dda-b318-bec55806e592-maas-api-service-ca\") pod \"86325738-3dcd-4dda-b318-bec55806e592\" (UID: \"86325738-3dcd-4dda-b318-bec55806e592\") " Sep 14 11:37:28.327016 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:28.326996 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86325738-3dcd-4dda-b318-bec55806e592-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "86325738-3dcd-4dda-b318-bec55806e592" (UID: "86325738-3dcd-4dda-b318-bec55806e592"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:37:28.427461 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:28.427406 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86325738-3dcd-4dda-b318-bec55806e592-maas-api-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:37:29.164769 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:29.164742 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3f02a5-8nmlj" Sep 14 11:37:29.164943 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:29.164787 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3f02a5-8nmlj" event={"ID":"86325738-3dcd-4dda-b318-bec55806e592","Type":"ContainerDied","Data":"e1fb25572b9ee9bb4f5b05b0d9f61efb86e9727f68e057e733f52b951ba992f7"} Sep 14 11:37:29.164943 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:29.164842 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1fb25572b9ee9bb4f5b05b0d9f61efb86e9727f68e057e733f52b951ba992f7" Sep 14 11:37:30.393785 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.393758 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:37:30.442971 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.442897 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf338544-9d39-4e9e-9668-ba204a6e2fde-istio-podinfo\") pod \"cf338544-9d39-4e9e-9668-ba204a6e2fde\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " Sep 14 11:37:30.442971 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.442960 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-istio-data\") pod \"cf338544-9d39-4e9e-9668-ba204a6e2fde\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " Sep 14 11:37:30.443207 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.442980 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-istio-envoy\") pod \"cf338544-9d39-4e9e-9668-ba204a6e2fde\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " Sep 14 11:37:30.443207 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.443104 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf338544-9d39-4e9e-9668-ba204a6e2fde-istio-token\") pod \"cf338544-9d39-4e9e-9668-ba204a6e2fde\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " Sep 14 11:37:30.443207 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.443168 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-workload-socket\") pod \"cf338544-9d39-4e9e-9668-ba204a6e2fde\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " Sep 14 11:37:30.443207 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.443203 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-credential-socket\") pod \"cf338544-9d39-4e9e-9668-ba204a6e2fde\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " Sep 14 11:37:30.443421 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.443227 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf338544-9d39-4e9e-9668-ba204a6e2fde-istiod-ca-cert\") pod \"cf338544-9d39-4e9e-9668-ba204a6e2fde\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " Sep 14 11:37:30.443421 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.443255 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrhnw\" (UniqueName: \"kubernetes.io/projected/cf338544-9d39-4e9e-9668-ba204a6e2fde-kube-api-access-jrhnw\") pod \"cf338544-9d39-4e9e-9668-ba204a6e2fde\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " Sep 14 11:37:30.443421 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.443295 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-workload-certs\") pod \"cf338544-9d39-4e9e-9668-ba204a6e2fde\" (UID: \"cf338544-9d39-4e9e-9668-ba204a6e2fde\") " Sep 14 11:37:30.443421 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.443341 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-istio-data" (OuterVolumeSpecName: "istio-data") pod "cf338544-9d39-4e9e-9668-ba204a6e2fde" (UID: "cf338544-9d39-4e9e-9668-ba204a6e2fde"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:37:30.443619 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.443591 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:37:30.443619 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.443601 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf338544-9d39-4e9e-9668-ba204a6e2fde-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cf338544-9d39-4e9e-9668-ba204a6e2fde" (UID: "cf338544-9d39-4e9e-9668-ba204a6e2fde"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:37:30.444169 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.444127 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cf338544-9d39-4e9e-9668-ba204a6e2fde" (UID: "cf338544-9d39-4e9e-9668-ba204a6e2fde"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:37:30.444282 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.444209 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cf338544-9d39-4e9e-9668-ba204a6e2fde" (UID: "cf338544-9d39-4e9e-9668-ba204a6e2fde"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:37:30.444282 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.444241 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cf338544-9d39-4e9e-9668-ba204a6e2fde" (UID: "cf338544-9d39-4e9e-9668-ba204a6e2fde"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:37:30.445326 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.445298 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cf338544-9d39-4e9e-9668-ba204a6e2fde" (UID: "cf338544-9d39-4e9e-9668-ba204a6e2fde"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:37:30.445409 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.445340 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf338544-9d39-4e9e-9668-ba204a6e2fde-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cf338544-9d39-4e9e-9668-ba204a6e2fde" (UID: "cf338544-9d39-4e9e-9668-ba204a6e2fde"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 11:37:30.445409 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.445349 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf338544-9d39-4e9e-9668-ba204a6e2fde-istio-token" (OuterVolumeSpecName: "istio-token") pod "cf338544-9d39-4e9e-9668-ba204a6e2fde" (UID: "cf338544-9d39-4e9e-9668-ba204a6e2fde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:37:30.445409 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.445368 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf338544-9d39-4e9e-9668-ba204a6e2fde-kube-api-access-jrhnw" (OuterVolumeSpecName: "kube-api-access-jrhnw") pod "cf338544-9d39-4e9e-9668-ba204a6e2fde" (UID: "cf338544-9d39-4e9e-9668-ba204a6e2fde"). InnerVolumeSpecName "kube-api-access-jrhnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:37:30.544808 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.544779 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:37:30.544808 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.544804 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf338544-9d39-4e9e-9668-ba204a6e2fde-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:37:30.544808 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.544812 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:37:30.545021 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.544820 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:37:30.545021 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.544830 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf338544-9d39-4e9e-9668-ba204a6e2fde-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:37:30.545021 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.544837 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrhnw\" (UniqueName: \"kubernetes.io/projected/cf338544-9d39-4e9e-9668-ba204a6e2fde-kube-api-access-jrhnw\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:37:30.545021 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.544845 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf338544-9d39-4e9e-9668-ba204a6e2fde-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:37:30.545021 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:30.544852 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf338544-9d39-4e9e-9668-ba204a6e2fde-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:37:31.174980 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:31.174949 2556 generic.go:356] "Generic (PLEG): container finished" podID="cf338544-9d39-4e9e-9668-ba204a6e2fde" containerID="b8df8986d3a73c2abfb3fc1721a11a73b8935b8dac6693bbce9dbf4c8eea9e21" exitCode=0 Sep 14 11:37:31.175177 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:31.175010 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" Sep 14 11:37:31.175177 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:31.175027 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" event={"ID":"cf338544-9d39-4e9e-9668-ba204a6e2fde","Type":"ContainerDied","Data":"b8df8986d3a73c2abfb3fc1721a11a73b8935b8dac6693bbce9dbf4c8eea9e21"} Sep 14 11:37:31.175177 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:31.175129 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8" event={"ID":"cf338544-9d39-4e9e-9668-ba204a6e2fde","Type":"ContainerDied","Data":"23d180ec6546afa957f529954517aee388981dfc7c3f7f7ed0ddecf8b6b05f1d"} Sep 14 11:37:31.175177 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:31.175156 2556 scope.go:117] "RemoveContainer" containerID="b8df8986d3a73c2abfb3fc1721a11a73b8935b8dac6693bbce9dbf4c8eea9e21" Sep 14 11:37:31.186784 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:31.186741 2556 scope.go:117] "RemoveContainer" containerID="c3b1e079d480f6c5be8d4406a687d93be3d88adf6339e29c175cae718a1b46d1" Sep 14 11:37:31.194148 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:31.194126 2556 scope.go:117] "RemoveContainer" containerID="b8df8986d3a73c2abfb3fc1721a11a73b8935b8dac6693bbce9dbf4c8eea9e21" Sep 14 11:37:31.194397 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:37:31.194380 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8df8986d3a73c2abfb3fc1721a11a73b8935b8dac6693bbce9dbf4c8eea9e21\": container with ID starting with b8df8986d3a73c2abfb3fc1721a11a73b8935b8dac6693bbce9dbf4c8eea9e21 not found: ID does not exist" containerID="b8df8986d3a73c2abfb3fc1721a11a73b8935b8dac6693bbce9dbf4c8eea9e21" Sep 14 11:37:31.194473 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:31.194407 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8df8986d3a73c2abfb3fc1721a11a73b8935b8dac6693bbce9dbf4c8eea9e21"} err="failed to get container status \"b8df8986d3a73c2abfb3fc1721a11a73b8935b8dac6693bbce9dbf4c8eea9e21\": rpc error: code = NotFound desc = could not find container \"b8df8986d3a73c2abfb3fc1721a11a73b8935b8dac6693bbce9dbf4c8eea9e21\": container with ID starting with b8df8986d3a73c2abfb3fc1721a11a73b8935b8dac6693bbce9dbf4c8eea9e21 not found: ID does not exist" Sep 14 11:37:31.194473 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:31.194429 2556 scope.go:117] "RemoveContainer" containerID="c3b1e079d480f6c5be8d4406a687d93be3d88adf6339e29c175cae718a1b46d1" Sep 14 11:37:31.194666 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:37:31.194649 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3b1e079d480f6c5be8d4406a687d93be3d88adf6339e29c175cae718a1b46d1\": container with ID starting with c3b1e079d480f6c5be8d4406a687d93be3d88adf6339e29c175cae718a1b46d1 not found: ID does not exist" containerID="c3b1e079d480f6c5be8d4406a687d93be3d88adf6339e29c175cae718a1b46d1" Sep 14 11:37:31.194709 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:31.194672 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3b1e079d480f6c5be8d4406a687d93be3d88adf6339e29c175cae718a1b46d1"} err="failed to get container status \"c3b1e079d480f6c5be8d4406a687d93be3d88adf6339e29c175cae718a1b46d1\": rpc error: code = NotFound desc = could not find container \"c3b1e079d480f6c5be8d4406a687d93be3d88adf6339e29c175cae718a1b46d1\": container with ID starting with c3b1e079d480f6c5be8d4406a687d93be3d88adf6339e29c175cae718a1b46d1 not found: ID does not exist" Sep 14 11:37:31.201361 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:31.201340 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8"] Sep 14 11:37:31.205194 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:31.205173 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-55c025-openshift-default-5f44cf86bd-hmvk8"] Sep 14 11:37:32.382180 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:32.382147 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf338544-9d39-4e9e-9668-ba204a6e2fde" path="/var/lib/kubelet/pods/cf338544-9d39-4e9e-9668-ba204a6e2fde/volumes" Sep 14 11:37:35.991791 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:35.991760 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3f02a5-29823090-p9j4q"] Sep 14 11:37:35.994185 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:35.994163 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3f02a5-29823090-p9j4q"] Sep 14 11:37:36.169981 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:36.169946 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3f02a5-8nmlj"] Sep 14 11:37:36.172048 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:36.172021 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3f02a5-8nmlj"] Sep 14 11:37:36.383247 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:36.383206 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86325738-3dcd-4dda-b318-bec55806e592" path="/var/lib/kubelet/pods/86325738-3dcd-4dda-b318-bec55806e592/volumes" Sep 14 11:37:36.383632 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:36.383613 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a78120fc-2b21-419f-90b7-e43c3841861b" path="/var/lib/kubelet/pods/a78120fc-2b21-419f-90b7-e43c3841861b/volumes" Sep 14 11:37:42.893186 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:42.893151 2556 scope.go:117] "RemoveContainer" containerID="6ddd9b859084c69c7a9e10b5cadd840781a556fd5c608170096b3099c25f5150" Sep 14 11:37:42.903766 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:42.903744 2556 scope.go:117] "RemoveContainer" containerID="5ce39195c1df6630dfe38afdc002f216a28512f52ab22efbaf42e5397a7c851a" Sep 14 11:37:42.915874 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:42.915851 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p"] Sep 14 11:37:42.916127 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:42.916070 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" podUID="11f0f8a3-1879-470f-aedc-5a6dfb04e3bc" containerName="istio-proxy" containerID="cri-o://d965dd94ae4706c76a3640c8666f474d11d2a303bfadfd0b2e0ee7cfb41d67a9" gracePeriod=30 Sep 14 11:37:43.048394 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:43.048358 2556 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 14 11:37:43.048556 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:43.048427 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" podUID="11f0f8a3-1879-470f-aedc-5a6dfb04e3bc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 14 11:37:48.171249 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.171224 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:37:48.275397 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.275308 2556 generic.go:356] "Generic (PLEG): container finished" podID="11f0f8a3-1879-470f-aedc-5a6dfb04e3bc" containerID="d965dd94ae4706c76a3640c8666f474d11d2a303bfadfd0b2e0ee7cfb41d67a9" exitCode=0 Sep 14 11:37:48.275397 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.275354 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" Sep 14 11:37:48.275576 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.275394 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" event={"ID":"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc","Type":"ContainerDied","Data":"d965dd94ae4706c76a3640c8666f474d11d2a303bfadfd0b2e0ee7cfb41d67a9"} Sep 14 11:37:48.275576 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.275467 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p" event={"ID":"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc","Type":"ContainerDied","Data":"9dfbf2f47b2bf19f1df2640c42638701601968799e4fc7bb8d444af0dd21bb44"} Sep 14 11:37:48.275576 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.275488 2556 scope.go:117] "RemoveContainer" containerID="d965dd94ae4706c76a3640c8666f474d11d2a303bfadfd0b2e0ee7cfb41d67a9" Sep 14 11:37:48.283563 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.283544 2556 scope.go:117] "RemoveContainer" containerID="f8ec02d95902757b3defae47a62f71116aebc0cef72a416ec34019b165815878" Sep 14 11:37:48.289748 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.289727 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istio-envoy\") pod \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " Sep 14 11:37:48.289856 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.289761 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istiod-ca-cert\") pod \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " Sep 14 11:37:48.289856 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.289802 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-workload-certs\") pod \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " Sep 14 11:37:48.289856 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.289819 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-workload-socket\") pod \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " Sep 14 11:37:48.290035 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.289841 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-credential-socket\") pod \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " Sep 14 11:37:48.290035 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.289916 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istio-podinfo\") pod \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " Sep 14 11:37:48.290035 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.289972 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istio-data\") pod \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " Sep 14 11:37:48.290035 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.290015 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istio-token\") pod \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " Sep 14 11:37:48.290257 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.290065 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p286j\" (UniqueName: \"kubernetes.io/projected/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-kube-api-access-p286j\") pod \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\" (UID: \"11f0f8a3-1879-470f-aedc-5a6dfb04e3bc\") " Sep 14 11:37:48.290535 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.290303 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "11f0f8a3-1879-470f-aedc-5a6dfb04e3bc" (UID: "11f0f8a3-1879-470f-aedc-5a6dfb04e3bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:37:48.290535 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.290429 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "11f0f8a3-1879-470f-aedc-5a6dfb04e3bc" (UID: "11f0f8a3-1879-470f-aedc-5a6dfb04e3bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:37:48.290682 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.290512 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "11f0f8a3-1879-470f-aedc-5a6dfb04e3bc" (UID: "11f0f8a3-1879-470f-aedc-5a6dfb04e3bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:37:48.290794 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.290763 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istio-data" (OuterVolumeSpecName: "istio-data") pod "11f0f8a3-1879-470f-aedc-5a6dfb04e3bc" (UID: "11f0f8a3-1879-470f-aedc-5a6dfb04e3bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:37:48.290852 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.290798 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "11f0f8a3-1879-470f-aedc-5a6dfb04e3bc" (UID: "11f0f8a3-1879-470f-aedc-5a6dfb04e3bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:37:48.292572 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.292535 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istio-token" (OuterVolumeSpecName: "istio-token") pod "11f0f8a3-1879-470f-aedc-5a6dfb04e3bc" (UID: "11f0f8a3-1879-470f-aedc-5a6dfb04e3bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:37:48.292668 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.292604 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "11f0f8a3-1879-470f-aedc-5a6dfb04e3bc" (UID: "11f0f8a3-1879-470f-aedc-5a6dfb04e3bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 11:37:48.292668 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.292638 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "11f0f8a3-1879-470f-aedc-5a6dfb04e3bc" (UID: "11f0f8a3-1879-470f-aedc-5a6dfb04e3bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:37:48.293522 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.293496 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-kube-api-access-p286j" (OuterVolumeSpecName: "kube-api-access-p286j") pod "11f0f8a3-1879-470f-aedc-5a6dfb04e3bc" (UID: "11f0f8a3-1879-470f-aedc-5a6dfb04e3bc"). InnerVolumeSpecName "kube-api-access-p286j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:37:48.293522 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.293515 2556 scope.go:117] "RemoveContainer" containerID="d965dd94ae4706c76a3640c8666f474d11d2a303bfadfd0b2e0ee7cfb41d67a9" Sep 14 11:37:48.293873 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:37:48.293845 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d965dd94ae4706c76a3640c8666f474d11d2a303bfadfd0b2e0ee7cfb41d67a9\": container with ID starting with d965dd94ae4706c76a3640c8666f474d11d2a303bfadfd0b2e0ee7cfb41d67a9 not found: ID does not exist" containerID="d965dd94ae4706c76a3640c8666f474d11d2a303bfadfd0b2e0ee7cfb41d67a9" Sep 14 11:37:48.293930 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.293879 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d965dd94ae4706c76a3640c8666f474d11d2a303bfadfd0b2e0ee7cfb41d67a9"} err="failed to get container status \"d965dd94ae4706c76a3640c8666f474d11d2a303bfadfd0b2e0ee7cfb41d67a9\": rpc error: code = NotFound desc = could not find container \"d965dd94ae4706c76a3640c8666f474d11d2a303bfadfd0b2e0ee7cfb41d67a9\": container with ID starting with d965dd94ae4706c76a3640c8666f474d11d2a303bfadfd0b2e0ee7cfb41d67a9 not found: ID does not exist" Sep 14 11:37:48.293930 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.293903 2556 scope.go:117] "RemoveContainer" containerID="f8ec02d95902757b3defae47a62f71116aebc0cef72a416ec34019b165815878" Sep 14 11:37:48.294251 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:37:48.294225 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8ec02d95902757b3defae47a62f71116aebc0cef72a416ec34019b165815878\": container with ID starting with f8ec02d95902757b3defae47a62f71116aebc0cef72a416ec34019b165815878 not found: ID does not exist" containerID="f8ec02d95902757b3defae47a62f71116aebc0cef72a416ec34019b165815878" Sep 14 11:37:48.294322 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.294258 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ec02d95902757b3defae47a62f71116aebc0cef72a416ec34019b165815878"} err="failed to get container status \"f8ec02d95902757b3defae47a62f71116aebc0cef72a416ec34019b165815878\": rpc error: code = NotFound desc = could not find container \"f8ec02d95902757b3defae47a62f71116aebc0cef72a416ec34019b165815878\": container with ID starting with f8ec02d95902757b3defae47a62f71116aebc0cef72a416ec34019b165815878 not found: ID does not exist" Sep 14 11:37:48.390770 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.390742 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istio-token\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:37:48.390879 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.390772 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p286j\" (UniqueName: \"kubernetes.io/projected/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-kube-api-access-p286j\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:37:48.390879 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.390787 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istio-envoy\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:37:48.390879 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.390802 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istiod-ca-cert\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:37:48.390879 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.390815 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-workload-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:37:48.390879 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.390827 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-workload-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:37:48.390879 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.390839 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-credential-socket\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:37:48.390879 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.390851 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istio-podinfo\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:37:48.390879 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.390863 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc-istio-data\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:37:48.591785 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.591688 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p"] Sep 14 11:37:48.594777 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:48.594743 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3f02a5-openshift-default-79d58dbb56-4kx9p"] Sep 14 11:37:50.382175 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:37:50.382144 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11f0f8a3-1879-470f-aedc-5a6dfb04e3bc" path="/var/lib/kubelet/pods/11f0f8a3-1879-470f-aedc-5a6dfb04e3bc/volumes" Sep 14 11:40:26.548562 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:40:26.548528 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65"] Sep 14 11:40:26.548991 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:40:26.548743 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65" podUID="32e5fce6-40e2-4456-bc82-d258c4ccf5d1" containerName="manager" containerID="cri-o://3928b2e0e8b0e8d992e694ce50d2186981fe89ae9d2aa3690585dbe5c726765f" gracePeriod=10 Sep 14 11:40:26.984597 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:40:26.984562 2556 generic.go:356] "Generic (PLEG): container finished" podID="32e5fce6-40e2-4456-bc82-d258c4ccf5d1" containerID="3928b2e0e8b0e8d992e694ce50d2186981fe89ae9d2aa3690585dbe5c726765f" exitCode=0 Sep 14 11:40:26.984770 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:40:26.984635 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65" event={"ID":"32e5fce6-40e2-4456-bc82-d258c4ccf5d1","Type":"ContainerDied","Data":"3928b2e0e8b0e8d992e694ce50d2186981fe89ae9d2aa3690585dbe5c726765f"} Sep 14 11:40:26.984770 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:40:26.984739 2556 scope.go:117] "RemoveContainer" containerID="097c92e82e2c091f9d4d774626a8d5455ce600c3c207d340342f0075419db0df" Sep 14 11:40:27.287506 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:40:27.287482 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65" Sep 14 11:40:27.384255 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:40:27.384222 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbbzx\" (UniqueName: \"kubernetes.io/projected/32e5fce6-40e2-4456-bc82-d258c4ccf5d1-kube-api-access-bbbzx\") pod \"32e5fce6-40e2-4456-bc82-d258c4ccf5d1\" (UID: \"32e5fce6-40e2-4456-bc82-d258c4ccf5d1\") " Sep 14 11:40:27.384440 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:40:27.384262 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32e5fce6-40e2-4456-bc82-d258c4ccf5d1-extensions-socket-volume\") pod \"32e5fce6-40e2-4456-bc82-d258c4ccf5d1\" (UID: \"32e5fce6-40e2-4456-bc82-d258c4ccf5d1\") " Sep 14 11:40:27.384652 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:40:27.384629 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e5fce6-40e2-4456-bc82-d258c4ccf5d1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "32e5fce6-40e2-4456-bc82-d258c4ccf5d1" (UID: "32e5fce6-40e2-4456-bc82-d258c4ccf5d1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:40:27.386145 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:40:27.386122 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e5fce6-40e2-4456-bc82-d258c4ccf5d1-kube-api-access-bbbzx" (OuterVolumeSpecName: "kube-api-access-bbbzx") pod "32e5fce6-40e2-4456-bc82-d258c4ccf5d1" (UID: "32e5fce6-40e2-4456-bc82-d258c4ccf5d1"). InnerVolumeSpecName "kube-api-access-bbbzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:40:27.485227 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:40:27.485201 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbbzx\" (UniqueName: \"kubernetes.io/projected/32e5fce6-40e2-4456-bc82-d258c4ccf5d1-kube-api-access-bbbzx\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:40:27.485227 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:40:27.485222 2556 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32e5fce6-40e2-4456-bc82-d258c4ccf5d1-extensions-socket-volume\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:40:27.990678 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:40:27.990647 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65" Sep 14 11:40:27.991070 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:40:27.990693 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65" event={"ID":"32e5fce6-40e2-4456-bc82-d258c4ccf5d1","Type":"ContainerDied","Data":"a4f57bb8b21ef29d58ddb61602ccbb122fe7e17eadce2a8a258d34ef06499402"} Sep 14 11:40:27.991070 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:40:27.990746 2556 scope.go:117] "RemoveContainer" containerID="3928b2e0e8b0e8d992e694ce50d2186981fe89ae9d2aa3690585dbe5c726765f" Sep 14 11:40:28.011307 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:40:28.011280 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65"] Sep 14 11:40:28.015822 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:40:28.015800 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5bn65"] Sep 14 11:40:28.381655 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:40:28.381626 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32e5fce6-40e2-4456-bc82-d258c4ccf5d1" path="/var/lib/kubelet/pods/32e5fce6-40e2-4456-bc82-d258c4ccf5d1/volumes" Sep 14 11:41:27.206784 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.206748 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-slsrn"] Sep 14 11:41:27.207254 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.207019 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11f0f8a3-1879-470f-aedc-5a6dfb04e3bc" containerName="istio-proxy" Sep 14 11:41:27.207254 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.207029 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="11f0f8a3-1879-470f-aedc-5a6dfb04e3bc" containerName="istio-proxy" Sep 14 11:41:27.207254 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.207039 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf338544-9d39-4e9e-9668-ba204a6e2fde" containerName="istio-proxy" Sep 14 11:41:27.207254 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.207044 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf338544-9d39-4e9e-9668-ba204a6e2fde" containerName="istio-proxy" Sep 14 11:41:27.207254 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.207052 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32e5fce6-40e2-4456-bc82-d258c4ccf5d1" containerName="manager" Sep 14 11:41:27.207254 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.207058 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="32e5fce6-40e2-4456-bc82-d258c4ccf5d1" containerName="manager" Sep 14 11:41:27.207254 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.207069 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32e5fce6-40e2-4456-bc82-d258c4ccf5d1" containerName="manager" Sep 14 11:41:27.207254 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.207074 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="32e5fce6-40e2-4456-bc82-d258c4ccf5d1" containerName="manager" Sep 14 11:41:27.207254 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.207104 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86325738-3dcd-4dda-b318-bec55806e592" containerName="revoke-keys" Sep 14 11:41:27.207254 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.207110 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="86325738-3dcd-4dda-b318-bec55806e592" containerName="revoke-keys" Sep 14 11:41:27.207254 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.207117 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf338544-9d39-4e9e-9668-ba204a6e2fde" containerName="istio-proxy" Sep 14 11:41:27.207254 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.207121 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf338544-9d39-4e9e-9668-ba204a6e2fde" containerName="istio-proxy" Sep 14 11:41:27.207254 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.207170 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="86325738-3dcd-4dda-b318-bec55806e592" containerName="revoke-keys" Sep 14 11:41:27.207254 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.207178 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="11f0f8a3-1879-470f-aedc-5a6dfb04e3bc" containerName="istio-proxy" Sep 14 11:41:27.207254 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.207184 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="32e5fce6-40e2-4456-bc82-d258c4ccf5d1" containerName="manager" Sep 14 11:41:27.207254 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.207189 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf338544-9d39-4e9e-9668-ba204a6e2fde" containerName="istio-proxy" Sep 14 11:41:27.207254 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.207193 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf338544-9d39-4e9e-9668-ba204a6e2fde" containerName="istio-proxy" Sep 14 11:41:27.210008 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.209992 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-slsrn" Sep 14 11:41:27.212452 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.212431 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-25c6l\"" Sep 14 11:41:27.220818 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.220795 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-slsrn"] Sep 14 11:41:27.284137 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.284108 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39c0a7ca-0bfa-433c-a242-8fffa4fdd752-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-slsrn\" (UID: \"39c0a7ca-0bfa-433c-a242-8fffa4fdd752\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-slsrn" Sep 14 11:41:27.284277 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.284138 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfb7r\" (UniqueName: \"kubernetes.io/projected/39c0a7ca-0bfa-433c-a242-8fffa4fdd752-kube-api-access-pfb7r\") pod \"kuadrant-operator-controller-manager-5769f99d9f-slsrn\" (UID: \"39c0a7ca-0bfa-433c-a242-8fffa4fdd752\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-slsrn" Sep 14 11:41:27.385330 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.385293 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39c0a7ca-0bfa-433c-a242-8fffa4fdd752-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-slsrn\" (UID: \"39c0a7ca-0bfa-433c-a242-8fffa4fdd752\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-slsrn" Sep 14 11:41:27.385330 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.385328 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfb7r\" (UniqueName: \"kubernetes.io/projected/39c0a7ca-0bfa-433c-a242-8fffa4fdd752-kube-api-access-pfb7r\") pod \"kuadrant-operator-controller-manager-5769f99d9f-slsrn\" (UID: \"39c0a7ca-0bfa-433c-a242-8fffa4fdd752\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-slsrn" Sep 14 11:41:27.385648 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.385628 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39c0a7ca-0bfa-433c-a242-8fffa4fdd752-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-slsrn\" (UID: \"39c0a7ca-0bfa-433c-a242-8fffa4fdd752\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-slsrn" Sep 14 11:41:27.394238 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.394215 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfb7r\" (UniqueName: \"kubernetes.io/projected/39c0a7ca-0bfa-433c-a242-8fffa4fdd752-kube-api-access-pfb7r\") pod \"kuadrant-operator-controller-manager-5769f99d9f-slsrn\" (UID: \"39c0a7ca-0bfa-433c-a242-8fffa4fdd752\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-slsrn" Sep 14 11:41:27.520070 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.519966 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-slsrn" Sep 14 11:41:27.795867 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:27.795699 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-slsrn"] Sep 14 11:41:27.798219 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:41:27.798191 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c0a7ca_0bfa_433c_a242_8fffa4fdd752.slice/crio-7dac6ee51805891a7039535a243c30c9ae945458c91e3430fcd0e95316e511a2 WatchSource:0}: Error finding container 7dac6ee51805891a7039535a243c30c9ae945458c91e3430fcd0e95316e511a2: Status 404 returned error can't find the container with id 7dac6ee51805891a7039535a243c30c9ae945458c91e3430fcd0e95316e511a2 Sep 14 11:41:28.250233 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:28.250199 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-slsrn" event={"ID":"39c0a7ca-0bfa-433c-a242-8fffa4fdd752","Type":"ContainerStarted","Data":"8390de7c978dff27e2b2fa616852618aff87ab5b272a1a5aa24721b274a5322a"} Sep 14 11:41:28.250623 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:28.250248 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-slsrn" event={"ID":"39c0a7ca-0bfa-433c-a242-8fffa4fdd752","Type":"ContainerStarted","Data":"7dac6ee51805891a7039535a243c30c9ae945458c91e3430fcd0e95316e511a2"} Sep 14 11:41:28.250623 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:28.250307 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-slsrn" Sep 14 11:41:28.274710 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:28.274660 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-slsrn" podStartSLOduration=1.274646819 podStartE2EDuration="1.274646819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:41:28.272720689 +0000 UTC m=+2446.336123601" watchObservedRunningTime="2026-09-14 11:41:28.274646819 +0000 UTC m=+2446.338049729" Sep 14 11:41:39.256951 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:39.256921 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-slsrn" Sep 14 11:41:42.972412 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:41:42.972378 2556 scope.go:117] "RemoveContainer" containerID="a43502a11b469cef48275ca7b54a0351c30e5cafd66c6fe568bb28c44ec37602" Sep 14 11:42:13.941777 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:13.941735 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6"] Sep 14 11:42:13.942367 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:13.942180 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11f0f8a3-1879-470f-aedc-5a6dfb04e3bc" containerName="istio-proxy" Sep 14 11:42:13.942367 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:13.942201 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="11f0f8a3-1879-470f-aedc-5a6dfb04e3bc" containerName="istio-proxy" Sep 14 11:42:13.942367 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:13.942286 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="32e5fce6-40e2-4456-bc82-d258c4ccf5d1" containerName="manager" Sep 14 11:42:13.942367 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:13.942302 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="11f0f8a3-1879-470f-aedc-5a6dfb04e3bc" containerName="istio-proxy" Sep 14 11:42:13.945745 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:13.945715 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:13.948285 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:13.948261 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c35b58a6-kserve-self-signed-certs\"" Sep 14 11:42:13.955923 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:13.955901 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6"] Sep 14 11:42:14.076844 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:14.076812 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-tls-certs\") pod \"e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6\" (UID: \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\") " pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:14.076844 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:14.076854 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgklm\" (UniqueName: \"kubernetes.io/projected/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-kube-api-access-cgklm\") pod \"e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6\" (UID: \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\") " pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:14.077174 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:14.076884 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-tmp-dir\") pod \"e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6\" (UID: \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\") " pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:14.077174 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:14.076906 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-home\") pod \"e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6\" (UID: \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\") " pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:14.077174 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:14.076961 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-dshm\") pod \"e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6\" (UID: \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\") " pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:14.077174 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:14.077013 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-model-cache\") pod \"e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6\" (UID: \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\") " pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:14.177696 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:14.177663 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-dshm\") pod \"e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6\" (UID: \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\") " pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:14.177696 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:14.177700 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-model-cache\") pod \"e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6\" (UID: \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\") " pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:14.177905 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:14.177754 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-tls-certs\") pod \"e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6\" (UID: \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\") " pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:14.177905 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:14.177871 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgklm\" (UniqueName: \"kubernetes.io/projected/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-kube-api-access-cgklm\") pod \"e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6\" (UID: \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\") " pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:14.178013 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:14.177917 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-tmp-dir\") pod \"e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6\" (UID: \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\") " pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:14.178013 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:14.177946 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-home\") pod \"e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6\" (UID: \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\") " pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:14.178214 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:14.178187 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-model-cache\") pod \"e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6\" (UID: \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\") " pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:14.178342 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:14.178313 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-tmp-dir\") pod \"e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6\" (UID: \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\") " pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:14.178389 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:14.178349 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-home\") pod \"e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6\" (UID: \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\") " pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:14.179995 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:14.179971 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-dshm\") pod \"e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6\" (UID: \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\") " pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:14.180211 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:14.180193 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-tls-certs\") pod \"e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6\" (UID: \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\") " pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:14.186363 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:14.186335 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgklm\" (UniqueName: \"kubernetes.io/projected/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-kube-api-access-cgklm\") pod \"e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6\" (UID: \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\") " pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:14.254952 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:14.254892 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:14.527691 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:14.527666 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6"] Sep 14 11:42:14.529548 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:42:14.529516 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf643964b_6f86_4b24_a41d_19eaa7c4c9f0.slice/crio-d6f1e52be27ca35c99d3a2d23089926eaac39ae74b3c125c474884169f32ff7b WatchSource:0}: Error finding container d6f1e52be27ca35c99d3a2d23089926eaac39ae74b3c125c474884169f32ff7b: Status 404 returned error can't find the container with id d6f1e52be27ca35c99d3a2d23089926eaac39ae74b3c125c474884169f32ff7b Sep 14 11:42:15.473458 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:15.473424 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" event={"ID":"f643964b-6f86-4b24-a41d-19eaa7c4c9f0","Type":"ContainerStarted","Data":"ce135040f0ec3094bff9707b7fdd6111d997f27bc3edf151ce9a873b127e3ad2"} Sep 14 11:42:15.473458 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:15.473465 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" event={"ID":"f643964b-6f86-4b24-a41d-19eaa7c4c9f0","Type":"ContainerStarted","Data":"d6f1e52be27ca35c99d3a2d23089926eaac39ae74b3c125c474884169f32ff7b"} Sep 14 11:42:15.510008 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:15.509956 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" podStartSLOduration=2.509942057 podStartE2EDuration="2.509942057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:42:15.509289039 +0000 UTC m=+2493.572691948" watchObservedRunningTime="2026-09-14 11:42:15.509942057 +0000 UTC m=+2493.573344963" Sep 14 11:42:24.255769 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:24.255735 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:24.256167 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:24.255786 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:24.268003 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:24.267978 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:24.526937 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:24.526859 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:41.477855 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:41.477785 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6"] Sep 14 11:42:42.990217 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:42.990182 2556 scope.go:117] "RemoveContainer" containerID="a800e48dc3c460ead0f71d3207eccae028d760ee93747e2c4ef9a1c990c53be6" Sep 14 11:42:56.496776 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:56.496733 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" podUID="f643964b-6f86-4b24-a41d-19eaa7c4c9f0" containerName="main" containerID="cri-o://ce135040f0ec3094bff9707b7fdd6111d997f27bc3edf151ce9a873b127e3ad2" gracePeriod=45 Sep 14 11:42:56.660042 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:56.660000 2556 generic.go:356] "Generic (PLEG): container finished" podID="f643964b-6f86-4b24-a41d-19eaa7c4c9f0" containerID="ce135040f0ec3094bff9707b7fdd6111d997f27bc3edf151ce9a873b127e3ad2" exitCode=0 Sep 14 11:42:56.660249 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:56.660050 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" event={"ID":"f643964b-6f86-4b24-a41d-19eaa7c4c9f0","Type":"ContainerDied","Data":"ce135040f0ec3094bff9707b7fdd6111d997f27bc3edf151ce9a873b127e3ad2"} Sep 14 11:42:56.737305 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:56.737282 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:56.841535 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:56.841453 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-tmp-dir\") pod \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\" (UID: \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\") " Sep 14 11:42:56.841535 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:56.841511 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgklm\" (UniqueName: \"kubernetes.io/projected/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-kube-api-access-cgklm\") pod \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\" (UID: \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\") " Sep 14 11:42:56.841535 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:56.841529 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-home\") pod \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\" (UID: \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\") " Sep 14 11:42:56.841829 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:56.841572 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-model-cache\") pod \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\" (UID: \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\") " Sep 14 11:42:56.841829 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:56.841600 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-tls-certs\") pod \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\" (UID: \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\") " Sep 14 11:42:56.841829 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:56.841632 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-dshm\") pod \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\" (UID: \"f643964b-6f86-4b24-a41d-19eaa7c4c9f0\") " Sep 14 11:42:56.841829 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:56.841768 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f643964b-6f86-4b24-a41d-19eaa7c4c9f0" (UID: "f643964b-6f86-4b24-a41d-19eaa7c4c9f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:42:56.841829 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:56.841813 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-home" (OuterVolumeSpecName: "home") pod "f643964b-6f86-4b24-a41d-19eaa7c4c9f0" (UID: "f643964b-6f86-4b24-a41d-19eaa7c4c9f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:42:56.841993 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:56.841828 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "f643964b-6f86-4b24-a41d-19eaa7c4c9f0" (UID: "f643964b-6f86-4b24-a41d-19eaa7c4c9f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:42:56.843733 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:56.843702 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-dshm" (OuterVolumeSpecName: "dshm") pod "f643964b-6f86-4b24-a41d-19eaa7c4c9f0" (UID: "f643964b-6f86-4b24-a41d-19eaa7c4c9f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:42:56.843823 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:56.843747 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f643964b-6f86-4b24-a41d-19eaa7c4c9f0" (UID: "f643964b-6f86-4b24-a41d-19eaa7c4c9f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:42:56.843823 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:56.843787 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-kube-api-access-cgklm" (OuterVolumeSpecName: "kube-api-access-cgklm") pod "f643964b-6f86-4b24-a41d-19eaa7c4c9f0" (UID: "f643964b-6f86-4b24-a41d-19eaa7c4c9f0"). InnerVolumeSpecName "kube-api-access-cgklm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:42:56.942387 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:56.942345 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgklm\" (UniqueName: \"kubernetes.io/projected/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-kube-api-access-cgklm\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:42:56.942387 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:56.942382 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-home\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:42:56.942387 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:56.942393 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-model-cache\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:42:56.942387 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:56.942402 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-tls-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:42:56.942651 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:56.942409 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-dshm\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:42:56.942651 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:56.942416 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f643964b-6f86-4b24-a41d-19eaa7c4c9f0-tmp-dir\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:42:57.666594 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:57.666565 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" Sep 14 11:42:57.666997 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:57.666571 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6" event={"ID":"f643964b-6f86-4b24-a41d-19eaa7c4c9f0","Type":"ContainerDied","Data":"d6f1e52be27ca35c99d3a2d23089926eaac39ae74b3c125c474884169f32ff7b"} Sep 14 11:42:57.666997 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:57.666697 2556 scope.go:117] "RemoveContainer" containerID="ce135040f0ec3094bff9707b7fdd6111d997f27bc3edf151ce9a873b127e3ad2" Sep 14 11:42:57.689691 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:57.689668 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6"] Sep 14 11:42:57.691590 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:57.691567 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c35b58a6-kserve-849845fd7f-k2hl6"] Sep 14 11:42:58.381842 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:42:58.381810 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f643964b-6f86-4b24-a41d-19eaa7c4c9f0" path="/var/lib/kubelet/pods/f643964b-6f86-4b24-a41d-19eaa7c4c9f0/volumes" Sep 14 11:43:04.797432 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:04.797396 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk"] Sep 14 11:43:04.797897 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:04.797683 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f643964b-6f86-4b24-a41d-19eaa7c4c9f0" containerName="main" Sep 14 11:43:04.797897 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:04.797694 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f643964b-6f86-4b24-a41d-19eaa7c4c9f0" containerName="main" Sep 14 11:43:04.797897 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:04.797750 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f643964b-6f86-4b24-a41d-19eaa7c4c9f0" containerName="main" Sep 14 11:43:04.801990 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:04.801968 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk" Sep 14 11:43:04.805604 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:04.805581 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w7q69\"" Sep 14 11:43:04.805739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:04.805600 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 14 11:43:04.805739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:04.805666 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 11:43:04.805739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:04.805668 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 14 11:43:04.805739 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:04.805712 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 14 11:43:04.818576 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:04.818555 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk"] Sep 14 11:43:04.908705 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:04.908669 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e60a241-f4cb-4b24-9b16-a8ee211daea8-maas-api-tls\") pod \"maas-api-8f99c9866-jtlkk\" (UID: \"1e60a241-f4cb-4b24-9b16-a8ee211daea8\") " pod="odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk" Sep 14 11:43:04.908705 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:04.908710 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e60a241-f4cb-4b24-9b16-a8ee211daea8-openshift-service-ca\") pod \"maas-api-8f99c9866-jtlkk\" (UID: \"1e60a241-f4cb-4b24-9b16-a8ee211daea8\") " pod="odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk" Sep 14 11:43:04.908935 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:04.908730 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh5cl\" (UniqueName: \"kubernetes.io/projected/1e60a241-f4cb-4b24-9b16-a8ee211daea8-kube-api-access-jh5cl\") pod \"maas-api-8f99c9866-jtlkk\" (UID: \"1e60a241-f4cb-4b24-9b16-a8ee211daea8\") " pod="odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk" Sep 14 11:43:04.908935 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:04.908815 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e60a241-f4cb-4b24-9b16-a8ee211daea8-metrics-certs\") pod \"maas-api-8f99c9866-jtlkk\" (UID: \"1e60a241-f4cb-4b24-9b16-a8ee211daea8\") " pod="odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk" Sep 14 11:43:05.009703 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:05.009672 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e60a241-f4cb-4b24-9b16-a8ee211daea8-maas-api-tls\") pod \"maas-api-8f99c9866-jtlkk\" (UID: \"1e60a241-f4cb-4b24-9b16-a8ee211daea8\") " pod="odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk" Sep 14 11:43:05.009857 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:05.009707 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e60a241-f4cb-4b24-9b16-a8ee211daea8-openshift-service-ca\") pod \"maas-api-8f99c9866-jtlkk\" (UID: \"1e60a241-f4cb-4b24-9b16-a8ee211daea8\") " pod="odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk" Sep 14 11:43:05.009857 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:05.009730 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh5cl\" (UniqueName: \"kubernetes.io/projected/1e60a241-f4cb-4b24-9b16-a8ee211daea8-kube-api-access-jh5cl\") pod \"maas-api-8f99c9866-jtlkk\" (UID: \"1e60a241-f4cb-4b24-9b16-a8ee211daea8\") " pod="odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk" Sep 14 11:43:05.009857 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:05.009761 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e60a241-f4cb-4b24-9b16-a8ee211daea8-metrics-certs\") pod \"maas-api-8f99c9866-jtlkk\" (UID: \"1e60a241-f4cb-4b24-9b16-a8ee211daea8\") " pod="odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk" Sep 14 11:43:05.010389 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:05.010364 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e60a241-f4cb-4b24-9b16-a8ee211daea8-openshift-service-ca\") pod \"maas-api-8f99c9866-jtlkk\" (UID: \"1e60a241-f4cb-4b24-9b16-a8ee211daea8\") " pod="odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk" Sep 14 11:43:05.012107 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:05.012059 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e60a241-f4cb-4b24-9b16-a8ee211daea8-metrics-certs\") pod \"maas-api-8f99c9866-jtlkk\" (UID: \"1e60a241-f4cb-4b24-9b16-a8ee211daea8\") " pod="odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk" Sep 14 11:43:05.012229 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:05.012141 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e60a241-f4cb-4b24-9b16-a8ee211daea8-maas-api-tls\") pod \"maas-api-8f99c9866-jtlkk\" (UID: \"1e60a241-f4cb-4b24-9b16-a8ee211daea8\") " pod="odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk" Sep 14 11:43:05.019004 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:05.018982 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh5cl\" (UniqueName: \"kubernetes.io/projected/1e60a241-f4cb-4b24-9b16-a8ee211daea8-kube-api-access-jh5cl\") pod \"maas-api-8f99c9866-jtlkk\" (UID: \"1e60a241-f4cb-4b24-9b16-a8ee211daea8\") " pod="odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk" Sep 14 11:43:05.112034 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:05.112009 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk" Sep 14 11:43:05.406901 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:05.406817 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk"] Sep 14 11:43:05.412312 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:43:05.412285 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e60a241_f4cb_4b24_9b16_a8ee211daea8.slice/crio-aae71cb386e92ce9d1c53b5151a393a1b56439c77e5a6d44d0d4936aa749eb7f WatchSource:0}: Error finding container aae71cb386e92ce9d1c53b5151a393a1b56439c77e5a6d44d0d4936aa749eb7f: Status 404 returned error can't find the container with id aae71cb386e92ce9d1c53b5151a393a1b56439c77e5a6d44d0d4936aa749eb7f Sep 14 11:43:05.413877 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:05.413858 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 11:43:05.703744 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:05.703667 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk" event={"ID":"1e60a241-f4cb-4b24-9b16-a8ee211daea8","Type":"ContainerStarted","Data":"aae71cb386e92ce9d1c53b5151a393a1b56439c77e5a6d44d0d4936aa749eb7f"} Sep 14 11:43:06.709528 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:06.709493 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk" event={"ID":"1e60a241-f4cb-4b24-9b16-a8ee211daea8","Type":"ContainerStarted","Data":"d5fff5a1c4937e85d6c44575d64ec3830153a48447225a3c3d0c02db803092e0"} Sep 14 11:43:06.709911 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:06.709627 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk" Sep 14 11:43:06.737973 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:06.737925 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk" podStartSLOduration=2.313168992 podStartE2EDuration="2.737912473s" totalImagesPullingTime="424.743481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:43:06.736277034 +0000 UTC m=+2544.799679943" watchObservedRunningTime="2026-09-14 11:43:06.737912473 +0000 UTC m=+2544.801315414" Sep 14 11:43:12.438227 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:12.438185 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk"] Sep 14 11:43:12.438716 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:12.438387 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk" podUID="1e60a241-f4cb-4b24-9b16-a8ee211daea8" containerName="maas-api" containerID="cri-o://d5fff5a1c4937e85d6c44575d64ec3830153a48447225a3c3d0c02db803092e0" gracePeriod=30 Sep 14 11:43:12.443467 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:12.443442 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk" Sep 14 11:43:12.679622 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:12.679598 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk" Sep 14 11:43:12.735601 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:12.735512 2556 generic.go:356] "Generic (PLEG): container finished" podID="1e60a241-f4cb-4b24-9b16-a8ee211daea8" containerID="d5fff5a1c4937e85d6c44575d64ec3830153a48447225a3c3d0c02db803092e0" exitCode=0 Sep 14 11:43:12.735601 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:12.735557 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk" Sep 14 11:43:12.735781 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:12.735594 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk" event={"ID":"1e60a241-f4cb-4b24-9b16-a8ee211daea8","Type":"ContainerDied","Data":"d5fff5a1c4937e85d6c44575d64ec3830153a48447225a3c3d0c02db803092e0"} Sep 14 11:43:12.735781 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:12.735647 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk" event={"ID":"1e60a241-f4cb-4b24-9b16-a8ee211daea8","Type":"ContainerDied","Data":"aae71cb386e92ce9d1c53b5151a393a1b56439c77e5a6d44d0d4936aa749eb7f"} Sep 14 11:43:12.735781 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:12.735677 2556 scope.go:117] "RemoveContainer" containerID="d5fff5a1c4937e85d6c44575d64ec3830153a48447225a3c3d0c02db803092e0" Sep 14 11:43:12.746767 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:12.746736 2556 scope.go:117] "RemoveContainer" containerID="d5fff5a1c4937e85d6c44575d64ec3830153a48447225a3c3d0c02db803092e0" Sep 14 11:43:12.747227 ip-10-0-143-52 kubenswrapper[2556]: E0914 11:43:12.747198 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5fff5a1c4937e85d6c44575d64ec3830153a48447225a3c3d0c02db803092e0\": container with ID starting with d5fff5a1c4937e85d6c44575d64ec3830153a48447225a3c3d0c02db803092e0 not found: ID does not exist" containerID="d5fff5a1c4937e85d6c44575d64ec3830153a48447225a3c3d0c02db803092e0" Sep 14 11:43:12.747377 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:12.747238 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5fff5a1c4937e85d6c44575d64ec3830153a48447225a3c3d0c02db803092e0"} err="failed to get container status \"d5fff5a1c4937e85d6c44575d64ec3830153a48447225a3c3d0c02db803092e0\": rpc error: code = NotFound desc = could not find container \"d5fff5a1c4937e85d6c44575d64ec3830153a48447225a3c3d0c02db803092e0\": container with ID starting with d5fff5a1c4937e85d6c44575d64ec3830153a48447225a3c3d0c02db803092e0 not found: ID does not exist" Sep 14 11:43:12.773962 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:12.773931 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e60a241-f4cb-4b24-9b16-a8ee211daea8-openshift-service-ca\") pod \"1e60a241-f4cb-4b24-9b16-a8ee211daea8\" (UID: \"1e60a241-f4cb-4b24-9b16-a8ee211daea8\") " Sep 14 11:43:12.774081 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:12.773976 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e60a241-f4cb-4b24-9b16-a8ee211daea8-metrics-certs\") pod \"1e60a241-f4cb-4b24-9b16-a8ee211daea8\" (UID: \"1e60a241-f4cb-4b24-9b16-a8ee211daea8\") " Sep 14 11:43:12.774081 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:12.774001 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e60a241-f4cb-4b24-9b16-a8ee211daea8-maas-api-tls\") pod \"1e60a241-f4cb-4b24-9b16-a8ee211daea8\" (UID: \"1e60a241-f4cb-4b24-9b16-a8ee211daea8\") " Sep 14 11:43:12.774081 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:12.774019 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jh5cl\" (UniqueName: \"kubernetes.io/projected/1e60a241-f4cb-4b24-9b16-a8ee211daea8-kube-api-access-jh5cl\") pod \"1e60a241-f4cb-4b24-9b16-a8ee211daea8\" (UID: \"1e60a241-f4cb-4b24-9b16-a8ee211daea8\") " Sep 14 11:43:12.774408 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:12.774385 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e60a241-f4cb-4b24-9b16-a8ee211daea8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1e60a241-f4cb-4b24-9b16-a8ee211daea8" (UID: "1e60a241-f4cb-4b24-9b16-a8ee211daea8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:43:12.776117 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:12.776081 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e60a241-f4cb-4b24-9b16-a8ee211daea8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1e60a241-f4cb-4b24-9b16-a8ee211daea8" (UID: "1e60a241-f4cb-4b24-9b16-a8ee211daea8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:43:12.776208 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:12.776147 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e60a241-f4cb-4b24-9b16-a8ee211daea8-kube-api-access-jh5cl" (OuterVolumeSpecName: "kube-api-access-jh5cl") pod "1e60a241-f4cb-4b24-9b16-a8ee211daea8" (UID: "1e60a241-f4cb-4b24-9b16-a8ee211daea8"). InnerVolumeSpecName "kube-api-access-jh5cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:43:12.776208 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:12.776186 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e60a241-f4cb-4b24-9b16-a8ee211daea8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1e60a241-f4cb-4b24-9b16-a8ee211daea8" (UID: "1e60a241-f4cb-4b24-9b16-a8ee211daea8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:43:12.875526 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:12.875492 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e60a241-f4cb-4b24-9b16-a8ee211daea8-openshift-service-ca\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:43:12.875526 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:12.875523 2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e60a241-f4cb-4b24-9b16-a8ee211daea8-metrics-certs\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:43:12.875526 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:12.875533 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e60a241-f4cb-4b24-9b16-a8ee211daea8-maas-api-tls\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:43:12.875763 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:12.875541 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh5cl\" (UniqueName: \"kubernetes.io/projected/1e60a241-f4cb-4b24-9b16-a8ee211daea8-kube-api-access-jh5cl\") on node \"ip-10-0-143-52.ec2.internal\" DevicePath \"\"" Sep 14 11:43:13.059035 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:13.059004 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk"] Sep 14 11:43:13.062730 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:13.062706 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f99c9866-jtlkk"] Sep 14 11:43:14.381770 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:14.381738 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e60a241-f4cb-4b24-9b16-a8ee211daea8" path="/var/lib/kubelet/pods/1e60a241-f4cb-4b24-9b16-a8ee211daea8/volumes" Sep 14 11:43:34.913724 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:34.913688 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-d66bbffcf-cd2bk_551680a3-3886-4d7a-8f07-1a5683b4b4d3/authorino/0.log" Sep 14 11:43:40.390860 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:40.390822 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pgqgl_e7de2f60-4aa0-4b18-badf-f63301c93d4d/manager/0.log" Sep 14 11:43:40.520456 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:40.520428 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-zcmgx_eb948054-387f-4074-94e5-c798756fe277/manager/0.log" Sep 14 11:43:41.155668 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:41.155636 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85645c65-r4zgx_7b80f3c5-2662-420e-b7b4-ef154cceee05/copy-manifests/0.log" Sep 14 11:43:41.165308 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:41.165285 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85645c65-r4zgx_7b80f3c5-2662-420e-b7b4-ef154cceee05/manager/0.log" Sep 14 11:43:41.877159 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:41.877134 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29823075-mvksl_da081176-ae4e-434f-a22b-c5652e07661a/cleanup/0.log" Sep 14 11:43:41.996714 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:41.996688 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29823090-6d5x2_3f49044a-70b7-4a12-8b02-a45f74ece418/cleanup/0.log" Sep 14 11:43:42.900195 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:42.900153 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-d66bbffcf-cd2bk_551680a3-3886-4d7a-8f07-1a5683b4b4d3/authorino/0.log" Sep 14 11:43:43.007582 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:43.007550 2556 scope.go:117] "RemoveContainer" containerID="2dd39d115dc439adb2e5b7e5667cc125385541e810951e04b1ab46cc96a54b2d" Sep 14 11:43:43.014978 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:43.014950 2556 scope.go:117] "RemoveContainer" containerID="4c8f99f42eaf969bb12b78000136d9904a589b58854634011856e4e36f821c2f" Sep 14 11:43:43.021566 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:43.021550 2556 scope.go:117] "RemoveContainer" containerID="6e9ac40727d324901d427a922f16c1739d42d272bfb338c2f1bc83568bab7f01" Sep 14 11:43:43.389520 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:43.389491 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-slsrn_39c0a7ca-0bfa-433c-a242-8fffa4fdd752/manager/0.log" Sep 14 11:43:44.010288 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:44.010256 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt_f6c85e1d-391a-4ff7-a402-9739d226bd4b/istio-proxy/0.log" Sep 14 11:43:44.457103 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:44.457058 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fdw2s_3afd8f26-9ef1-47db-ac05-349b80a4c6b1/istio-proxy/1.log" Sep 14 11:43:45.216433 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:45.216404 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-8zb2r_8e62bde0-fa42-4ac3-83b4-74e7bd0fec04/main/0.log" Sep 14 11:43:45.580486 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:45.580419 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-gggh4_7e6ad956-e4b7-4c58-b145-486a8e935797/main/0.log" Sep 14 11:43:45.943963 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:45.943935 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-v8sm8_dc132696-2458-49ad-8b2e-75edf35e07f0/main/0.log" Sep 14 11:43:51.839027 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:51.838994 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zxfzq/must-gather-22mnq"] Sep 14 11:43:51.839456 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:51.839301 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e60a241-f4cb-4b24-9b16-a8ee211daea8" containerName="maas-api" Sep 14 11:43:51.839456 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:51.839314 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e60a241-f4cb-4b24-9b16-a8ee211daea8" containerName="maas-api" Sep 14 11:43:51.839456 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:51.839363 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e60a241-f4cb-4b24-9b16-a8ee211daea8" containerName="maas-api" Sep 14 11:43:51.842312 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:51.842293 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zxfzq/must-gather-22mnq" Sep 14 11:43:51.845641 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:51.845600 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zxfzq\"/\"default-dockercfg-2g5tp\"" Sep 14 11:43:51.845749 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:51.845731 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zxfzq\"/\"openshift-service-ca.crt\"" Sep 14 11:43:51.846916 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:51.846899 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zxfzq\"/\"kube-root-ca.crt\"" Sep 14 11:43:51.861732 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:51.861710 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zxfzq/must-gather-22mnq"] Sep 14 11:43:52.005518 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:52.005485 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2a3cba11-6a3c-4e53-84b4-6aeeb6969828-must-gather-output\") pod \"must-gather-22mnq\" (UID: \"2a3cba11-6a3c-4e53-84b4-6aeeb6969828\") " pod="openshift-must-gather-zxfzq/must-gather-22mnq" Sep 14 11:43:52.005518 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:52.005518 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8wzq\" (UniqueName: \"kubernetes.io/projected/2a3cba11-6a3c-4e53-84b4-6aeeb6969828-kube-api-access-s8wzq\") pod \"must-gather-22mnq\" (UID: \"2a3cba11-6a3c-4e53-84b4-6aeeb6969828\") " pod="openshift-must-gather-zxfzq/must-gather-22mnq" Sep 14 11:43:52.106277 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:52.106198 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2a3cba11-6a3c-4e53-84b4-6aeeb6969828-must-gather-output\") pod \"must-gather-22mnq\" (UID: \"2a3cba11-6a3c-4e53-84b4-6aeeb6969828\") " pod="openshift-must-gather-zxfzq/must-gather-22mnq" Sep 14 11:43:52.106277 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:52.106238 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8wzq\" (UniqueName: \"kubernetes.io/projected/2a3cba11-6a3c-4e53-84b4-6aeeb6969828-kube-api-access-s8wzq\") pod \"must-gather-22mnq\" (UID: \"2a3cba11-6a3c-4e53-84b4-6aeeb6969828\") " pod="openshift-must-gather-zxfzq/must-gather-22mnq" Sep 14 11:43:52.106546 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:52.106528 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2a3cba11-6a3c-4e53-84b4-6aeeb6969828-must-gather-output\") pod \"must-gather-22mnq\" (UID: \"2a3cba11-6a3c-4e53-84b4-6aeeb6969828\") " pod="openshift-must-gather-zxfzq/must-gather-22mnq" Sep 14 11:43:52.119127 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:52.119078 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8wzq\" (UniqueName: \"kubernetes.io/projected/2a3cba11-6a3c-4e53-84b4-6aeeb6969828-kube-api-access-s8wzq\") pod \"must-gather-22mnq\" (UID: \"2a3cba11-6a3c-4e53-84b4-6aeeb6969828\") " pod="openshift-must-gather-zxfzq/must-gather-22mnq" Sep 14 11:43:52.151128 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:52.151103 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zxfzq/must-gather-22mnq" Sep 14 11:43:52.443992 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:52.443967 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zxfzq/must-gather-22mnq"] Sep 14 11:43:52.445751 ip-10-0-143-52 kubenswrapper[2556]: W0914 11:43:52.445726 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a3cba11_6a3c_4e53_84b4_6aeeb6969828.slice/crio-1043a7708cc7fb35b26a2354981e5da97b72c6aeaa9fbc6cd7bd7b467f398847 WatchSource:0}: Error finding container 1043a7708cc7fb35b26a2354981e5da97b72c6aeaa9fbc6cd7bd7b467f398847: Status 404 returned error can't find the container with id 1043a7708cc7fb35b26a2354981e5da97b72c6aeaa9fbc6cd7bd7b467f398847 Sep 14 11:43:52.913445 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:52.913405 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxfzq/must-gather-22mnq" event={"ID":"2a3cba11-6a3c-4e53-84b4-6aeeb6969828","Type":"ContainerStarted","Data":"1043a7708cc7fb35b26a2354981e5da97b72c6aeaa9fbc6cd7bd7b467f398847"} Sep 14 11:43:53.919360 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:53.919321 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxfzq/must-gather-22mnq" event={"ID":"2a3cba11-6a3c-4e53-84b4-6aeeb6969828","Type":"ContainerStarted","Data":"e23c987805695708c049945ff4df990358554d9f4891a41d6512eeb33ddd068a"} Sep 14 11:43:53.919360 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:53.919366 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxfzq/must-gather-22mnq" event={"ID":"2a3cba11-6a3c-4e53-84b4-6aeeb6969828","Type":"ContainerStarted","Data":"39221efd0ad5bd42ae26b253635b1e513c73f78306c69ad281bb8bdfff60d5f2"} Sep 14 11:43:53.955902 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:53.955851 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zxfzq/must-gather-22mnq" podStartSLOduration=2.114007743 podStartE2EDuration="2.955830559s" totalImagesPullingTime="841.822816ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:43:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:43:53.951724493 +0000 UTC m=+2592.015127403" watchObservedRunningTime="2026-09-14 11:43:53.955830559 +0000 UTC m=+2592.019233480" Sep 14 11:43:55.397531 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:55.397498 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-96bxc_6547f534-40c9-42c7-b118-37389bd1a650/global-pull-secret-syncer/0.log" Sep 14 11:43:55.835758 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:55.835683 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vtv8p_e19a55ba-0b57-429c-a111-7b199d668301/konnectivity-agent/0.log" Sep 14 11:43:56.089766 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:56.089683 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-52.ec2.internal_c127f793063e49cd83ca3a678e448bb8/haproxy/0.log" Sep 14 11:43:59.526320 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:59.526289 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-d66bbffcf-cd2bk_551680a3-3886-4d7a-8f07-1a5683b4b4d3/authorino/0.log" Sep 14 11:43:59.978056 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:43:59.978018 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-slsrn_39c0a7ca-0bfa-433c-a242-8fffa4fdd752/manager/0.log" Sep 14 11:44:03.054002 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:03.053969 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2kcvz_748fb694-37f5-47cb-9fad-4cc1e1d7421f/node-exporter/0.log" Sep 14 11:44:03.093547 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:03.093512 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2kcvz_748fb694-37f5-47cb-9fad-4cc1e1d7421f/kube-rbac-proxy/0.log" Sep 14 11:44:03.143282 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:03.143249 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2kcvz_748fb694-37f5-47cb-9fad-4cc1e1d7421f/init-textfile/0.log" Sep 14 11:44:05.069583 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:05.069496 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zxfzq/perf-node-gather-daemonset-nzlzn"] Sep 14 11:44:05.077623 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:05.077598 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-nzlzn" Sep 14 11:44:05.099697 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:05.099661 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zxfzq/perf-node-gather-daemonset-nzlzn"] Sep 14 11:44:05.254810 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:05.254770 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f29b2827-75e6-46ff-b035-b8a1b8a68682-sys\") pod \"perf-node-gather-daemonset-nzlzn\" (UID: \"f29b2827-75e6-46ff-b035-b8a1b8a68682\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-nzlzn" Sep 14 11:44:05.254997 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:05.254839 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92xtw\" (UniqueName: \"kubernetes.io/projected/f29b2827-75e6-46ff-b035-b8a1b8a68682-kube-api-access-92xtw\") pod \"perf-node-gather-daemonset-nzlzn\" (UID: \"f29b2827-75e6-46ff-b035-b8a1b8a68682\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-nzlzn" Sep 14 11:44:05.254997 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:05.254880 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f29b2827-75e6-46ff-b035-b8a1b8a68682-podres\") pod \"perf-node-gather-daemonset-nzlzn\" (UID: \"f29b2827-75e6-46ff-b035-b8a1b8a68682\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-nzlzn" Sep 14 11:44:05.254997 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:05.254907 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f29b2827-75e6-46ff-b035-b8a1b8a68682-proc\") pod \"perf-node-gather-daemonset-nzlzn\" (UID: \"f29b2827-75e6-46ff-b035-b8a1b8a68682\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-nzlzn" Sep 14 11:44:05.254997 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:05.254970 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f29b2827-75e6-46ff-b035-b8a1b8a68682-lib-modules\") pod \"perf-node-gather-daemonset-nzlzn\" (UID: \"f29b2827-75e6-46ff-b035-b8a1b8a68682\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-nzlzn" Sep 14 11:44:05.356122 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:05.356018 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f29b2827-75e6-46ff-b035-b8a1b8a68682-lib-modules\") pod \"perf-node-gather-daemonset-nzlzn\" (UID: \"f29b2827-75e6-46ff-b035-b8a1b8a68682\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-nzlzn" Sep 14 11:44:05.356122 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:05.356063 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f29b2827-75e6-46ff-b035-b8a1b8a68682-sys\") pod \"perf-node-gather-daemonset-nzlzn\" (UID: \"f29b2827-75e6-46ff-b035-b8a1b8a68682\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-nzlzn" Sep 14 11:44:05.356304 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:05.356161 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f29b2827-75e6-46ff-b035-b8a1b8a68682-sys\") pod \"perf-node-gather-daemonset-nzlzn\" (UID: \"f29b2827-75e6-46ff-b035-b8a1b8a68682\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-nzlzn" Sep 14 11:44:05.356304 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:05.356192 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92xtw\" (UniqueName: \"kubernetes.io/projected/f29b2827-75e6-46ff-b035-b8a1b8a68682-kube-api-access-92xtw\") pod \"perf-node-gather-daemonset-nzlzn\" (UID: \"f29b2827-75e6-46ff-b035-b8a1b8a68682\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-nzlzn" Sep 14 11:44:05.356304 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:05.356221 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f29b2827-75e6-46ff-b035-b8a1b8a68682-podres\") pod \"perf-node-gather-daemonset-nzlzn\" (UID: \"f29b2827-75e6-46ff-b035-b8a1b8a68682\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-nzlzn" Sep 14 11:44:05.356304 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:05.356239 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f29b2827-75e6-46ff-b035-b8a1b8a68682-proc\") pod \"perf-node-gather-daemonset-nzlzn\" (UID: \"f29b2827-75e6-46ff-b035-b8a1b8a68682\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-nzlzn" Sep 14 11:44:05.356304 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:05.356249 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f29b2827-75e6-46ff-b035-b8a1b8a68682-lib-modules\") pod \"perf-node-gather-daemonset-nzlzn\" (UID: \"f29b2827-75e6-46ff-b035-b8a1b8a68682\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-nzlzn" Sep 14 11:44:05.356304 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:05.356296 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f29b2827-75e6-46ff-b035-b8a1b8a68682-proc\") pod \"perf-node-gather-daemonset-nzlzn\" (UID: \"f29b2827-75e6-46ff-b035-b8a1b8a68682\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-nzlzn" Sep 14 11:44:05.356501 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:05.356489 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f29b2827-75e6-46ff-b035-b8a1b8a68682-podres\") pod \"perf-node-gather-daemonset-nzlzn\" (UID: \"f29b2827-75e6-46ff-b035-b8a1b8a68682\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-nzlzn" Sep 14 11:44:05.367081 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:05.367050 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92xtw\" (UniqueName: \"kubernetes.io/projected/f29b2827-75e6-46ff-b035-b8a1b8a68682-kube-api-access-92xtw\") pod \"perf-node-gather-daemonset-nzlzn\" (UID: \"f29b2827-75e6-46ff-b035-b8a1b8a68682\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-nzlzn" Sep 14 11:44:05.390293 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:05.390254 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-nzlzn" Sep 14 11:44:05.709065 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:05.708232 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zxfzq/perf-node-gather-daemonset-nzlzn"] Sep 14 11:44:05.995344 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:05.995249 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-nzlzn" event={"ID":"f29b2827-75e6-46ff-b035-b8a1b8a68682","Type":"ContainerStarted","Data":"59908d497e81809a713403d4d5441736bb456803695cbeb91af454df6dd08dda"} Sep 14 11:44:05.995344 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:05.995316 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-nzlzn" event={"ID":"f29b2827-75e6-46ff-b035-b8a1b8a68682","Type":"ContainerStarted","Data":"427ad8ef7eb53199faf9a9a9bb3710b3350fcd8f949e7c7f615d5c15c8671f75"} Sep 14 11:44:05.996067 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:05.996038 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-nzlzn" Sep 14 11:44:06.018390 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:06.018341 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-nzlzn" podStartSLOduration=1.01832568 podStartE2EDuration="1.01832568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:44:06.014978285 +0000 UTC m=+2604.078381208" watchObservedRunningTime="2026-09-14 11:44:06.01832568 +0000 UTC m=+2604.081728599" Sep 14 11:44:09.159316 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:09.159290 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k264k_e59abbc8-c9b6-45ec-ae03-a7b534ca02df/dns/0.log" Sep 14 11:44:09.214297 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:09.214250 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k264k_e59abbc8-c9b6-45ec-ae03-a7b534ca02df/kube-rbac-proxy/0.log" Sep 14 11:44:09.504441 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:09.504369 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mcxhz_2f816499-fb81-4f59-bb3f-986f34d9c571/dns-node-resolver/0.log" Sep 14 11:44:10.294821 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:10.294793 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5qhj6_27fdd2fa-1378-4960-bd86-fd1dd84516c3/node-ca/0.log" Sep 14 11:44:11.594607 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:11.594581 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt_f6c85e1d-391a-4ff7-a402-9739d226bd4b/istio-proxy/0.log" Sep 14 11:44:12.189651 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:12.189619 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fdw2s_3afd8f26-9ef1-47db-ac05-349b80a4c6b1/istio-proxy/0.log" Sep 14 11:44:12.192380 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:12.192360 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fdw2s_3afd8f26-9ef1-47db-ac05-349b80a4c6b1/istio-proxy/1.log" Sep 14 11:44:13.016516 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:13.016482 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-nzlzn" Sep 14 11:44:13.595497 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:13.595469 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nvm9v_9bd76a15-e9b6-402e-ad38-e1a3ab4a1a18/serve-healthcheck-canary/0.log" Sep 14 11:44:14.575358 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:14.575327 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-24mtb_b49b5ae9-93d6-4fa5-afa0-3c6b64766be4/kube-rbac-proxy/0.log" Sep 14 11:44:14.621132 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:14.621105 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-24mtb_b49b5ae9-93d6-4fa5-afa0-3c6b64766be4/exporter/0.log" Sep 14 11:44:14.711037 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:14.710952 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-24mtb_b49b5ae9-93d6-4fa5-afa0-3c6b64766be4/extractor/0.log" Sep 14 11:44:14.792214 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:14.792186 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt_f6c85e1d-391a-4ff7-a402-9739d226bd4b/istio-proxy/0.log" Sep 14 11:44:15.440263 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:15.440176 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fdw2s_3afd8f26-9ef1-47db-ac05-349b80a4c6b1/istio-proxy/0.log" Sep 14 11:44:15.443663 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:15.443640 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fdw2s_3afd8f26-9ef1-47db-ac05-349b80a4c6b1/istio-proxy/1.log" Sep 14 11:44:17.185496 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:17.185424 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-pr2zt_f6c85e1d-391a-4ff7-a402-9739d226bd4b/istio-proxy/0.log" Sep 14 11:44:17.761390 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:17.761351 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fdw2s_3afd8f26-9ef1-47db-ac05-349b80a4c6b1/istio-proxy/0.log" Sep 14 11:44:17.765348 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:17.765321 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fdw2s_3afd8f26-9ef1-47db-ac05-349b80a4c6b1/istio-proxy/1.log" Sep 14 11:44:18.203016 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:18.202971 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pgqgl_e7de2f60-4aa0-4b18-badf-f63301c93d4d/manager/0.log" Sep 14 11:44:18.532608 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:18.532530 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-zcmgx_eb948054-387f-4074-94e5-c798756fe277/manager/0.log" Sep 14 11:44:18.960286 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:18.960252 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85645c65-r4zgx_7b80f3c5-2662-420e-b7b4-ef154cceee05/manager/0.log" Sep 14 11:44:18.999218 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:18.999193 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85645c65-r4zgx_7b80f3c5-2662-420e-b7b4-ef154cceee05/copy-manifests/0.log" Sep 14 11:44:20.500950 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:20.500922 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-69cb796bd9-h7xm9_91c62911-b234-42ff-82c3-739cbd1f14c3/manager/0.log" Sep 14 11:44:20.651687 ip-10-0-143-52 kubenswrapper[2556]: I0914 11:44:20.651658 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-r7s7d_3002f8b0-31d8-4b91-a90e-c5d7ea43cfdb/openshift-lws-operator/0.log"